Step 2- Configure and start the LedgerBridge Connector
In this section:
Prerequisites
- LedgerBridge is running
- A valid certificate and private key. You can download the certificate and private key for the current session from the Tools > Certificate Tools page.
Downloadable JAR files
The LedgerBridge connector supports the import of data from various types of files. Before you begin, download the LedgerBridge Connector JAR file and the required Connector-Collector and Connector-Transformer files from the Gospel Dashboard > Downloads section.
Starting the LedgerBridge Connector
You can start LedgerBridge Connector either by defining the configuration in the application.properties file or by passing the parameters as command-line arguments.
Using application.properties
You can store configurable properties in an application.properties file and place it in the same directory as the JAR file. Each parameter is stored as a pair of strings, one storing the name of the parameter, and the other storing the value. For example, The parameters set from the command line have precedence over the properties file in the project directory.dataLocation=datafiles where, dataLocation is the parameter and datafiles is the value.
If you have defined the configuration in the application.properties file, in the command prompt, go to the location of the directory where the gospel-ledgerbridge-<version>.jar file is located, and run the following command:
java -jar ledgerbridge-connector-<version>.jar
The LedgerBridge Administrative Panel > Connector Builder page displays the configuration currently applied when starting the LedgerBridge Connector.
On this page, you can provide the required configuration and generate a configuration file using the Download Config option. Save the file to the LedgerBridge Connector directory as application.properties and restart the LedgerBridge Connector to apply the new configuration.
Pass parameters in the terminal as command-line arguments
To pass the parameters in the command line arguments, go to the location of the directory where the gospel-ledgerbridge-<version>.jar file is located, and run the following command:
java -jar ledgerbridge-connector-<version>.jar --dataLocation=<path/to/directory> --ledgerbridgeBaseUrl=<http://localhost:port-number>
The code example lists some configuration values required for starting the LedgerBridge Connector. You can add additional configuration values as described in the table below.
To see all the parameters you can list in the applicaiton.properties, execute:
java -jar ledgerbridge-connector-<version>.jar --show-params
To see only the mandatory parameters execute:
java -jar ledgerbridge-connector-<version>.jar --show-mandatory-params
Connector Builder Settings
The table lists and describes the configuration values you can set when starting the LedgerBridge Connector:
If you make changes to the configuration, you must restart the LedgerBridge Connector to apply the changes.
Main LedgerBridge Connector configuration
| Configuration Properties | Description | Value |
|---|---|---|
httpRequestTimeoutSeconds | Optional. The timeout time in seconds for the HTTP client that connects to the LedegerBridge | Default: 30 |
ledgerbridgeBaseUrl | Optional. The LedgerBridge base URL in the form <protocol>://<host>:<port> | Default: http://localhost:8090 |
recordEndpoint | Optional. The LedgerBridge record endpoint to which the LedgerBridge connector sends records | Default: /api/post |
| Optional. The LedgerBridge heartbeat endpoint to which the LedgerBridge connector sends a periodical request | Default: /api/heartbeat |
configEndpoint | Optional. The LedgerBridge endpoint the to which the LedgerBridge connector sends its configuration | Default: /api/config |
userAgent | Optional. The User-Agent header sent by the HTTP client to the LedgerBridge | Default: GospelCollector |
preserveSecret | Optional. When set to true, the LedgerBridge Connector attempts to load secret from the file system, based on the parameters passed. Otherwise, it generates a random secret. | Default: true |
sharedSecretFilename | Optional. The file containing the secret passphrase that is shared between Ledgerbridge and Ledgerbridge connector | Default: .ledgerbridge |
qualifiedPathToSharedSecretDirectory | Optional. The directory containing the secret passphrase that is shared between Ledgerbridge and Ledgerbridge connector | Default: . |
| Optional. The shared secret length in bytes | Default: 64 |
sharedSecretTokenValidityPeriodSeconds | Optional. The period (in seconds) after which a LedgerBridge connection-token becomes invalid | Default: 15 |
batchSize | Optional. The records are sent to the LedgerBridge in batches. Set the number of records to be sent in each batch. For example, if you set the value to a big number, more data is sent to the LedgerBridge, and if the value is set to a small number, frequent requests are made to the LedgerBridge for data. | Default: 10 |
backOffQueueSize | Optional. The size of the internal backoff queue. If an error is reported when processing data, the records are added to the backOff queue until the error is resolved. | Default: 100 |
batchTimeoutSeconds | Optional. The timeout after which a record batch gets consumed, even if it has not reached the batch size. | Default value: 1 |
metricPrintPeriodMinutes | Duration in minutes between each set of metrics that are reported in the console Set the value to zero (0) to disable the feature. | Default: 5 |
metricLogLevel | Records the rate of logged events by their logging level Ensure that the following properties are set in the application.properties to enable logging:
| Default: Options:
|
Configuration for File Collector
Configuration Properties | Description | Value |
|---|---|---|
dataLocation | Required. The directory where the source files are located. You can change the default (root) location by providing the path to the folder containing the source data files. | Default: For example, |
searchFilemask | Required. Set the file mask to specify the files to be processed and the files to be ignored. To import data from compressed files, enter the extension of the file. For example, .tar.gz, .tar, tar.bz2, or .zip. | Default: For example, |
moveAfterProcessing | Optional. If set to true, the processed files are moved to a sub-directory. | Default: false |
moveProcessedFilesTo | Optional. If the | Default: ./processed |
deleteAfterProcessing | Optional. If set to true, all the processed files are deleted. | Default: false |
renameProcessedFilesTo | Optional. By default, all processed files are renamed. Specify the suffix to append to the processed file name. | Default: .old |
charset | Specifies the character encoding used in the files to be imported. If the encoding value of the file is different from the encoding value of the system, the import may fail. | For example: UTF-8 |
metricPrintPeriodMinutes | Duration in minutes between each set of metrics that are reported in the console Set the value to zero ( | Default: 5 |
metricLogLevel | Records the rate of logged events by their logging level Ensure that the following properties are set in the application.properties to enable logging:
| Default: Options:
|
Configuration for API collector
Configuration Properties | Description | Value |
|---|---|---|
url | Required. The URL to the source file. | |
method | Required. The API method to use. | Options:
|
| Optional. The HTTP headers to pass to the API invocation. | Configure as:
|
pollingTime | Optional. The pause (in seconds) between each API call | Default: 30 |
metricPrintPeriodMinutes | Duration in minutes between each set of metrics that are reported in the console Set the value to zero ( | Default: 5 |
metricLogLevel | Records the rate of logged events by their logging level Ensure that the following properties are set in the application.properties to enable logging:
| Default: Options:
|
Configuration for the JDBC collector plugin
| Configuration Properties | Description | Value |
|---|---|---|
| Required. The SQL query to execute on the database to which the plugin is connected | |
jdbcUrl | Required. The URL of the JDBC database to which the plugin connects | |
jdbcUsername | Required. The database username used to connect to the database | |
jdbcPassword | Optional. The password of the database user (if required) | |
pollingTime | Optional. The interval (in seconds) between each SQL query that is executed against the database | Default: 10 |
fetchSize | Optional. Determines how many records to fetch at a time from the database for processing | Default: 10 |
metricPrintPeriodMinutes | Duration in minutes between each set of metrics that are reported in the console Set the value to zero ( | Default: 5 |
metricLogLevel | Records the rate of logged events by their logging level Ensure that the following properties are set in the application.properties to enable logging:
| Default: Options:
|
Configuration for CSV transformer
| Configuration Properties | Description | Value |
|---|---|---|
pathToDefinition | Optional. Enter the location of the directory where the properties file is located. By default, the properties file must be placed in the directory where the LedgerBridge connector, collector and transformer JAR files are located. When importing CSV files, the headers in the CSV file are automatically mapped to fields in the record and therefore does not require a definitions file. You can, however, provide mapping using the definitions.properties file or you can define the headers in the application.properties file, in the parameter. | Default: ./definitions.properties |
recordDefinitionId | Optional. The ID of the Gospel record definition, to which the data is imported The ID field must begin with a letter ([A-Za-z]) | |
customCsvDelimiter | Optional. Set a specific column delimiter. During import, the delimiter helps identify where each field begins and ends in the imported file. | Default: , |
customCsvRegex | Optional. Specify a custom behaviour when splitting columns. For example, you may want to consider the string "abc, def" (enclosed within double quotes) as a single column. | Default: (?=(?:[^\"]*\"[^\"]*\")*[^\"]*$) |
customCsvHeaderLine | Optional. Specify a custom header for the CSV files being imported. This is useful when the input files do not have a header line. For example: | |
allowFileUploads | Optional. Set to true, to allow files to be uploaded into a Gospel record field | Default: false |
maxFileSizeMB | Optional. The maximum size (in MB) for a file to be uploaded to a field in a record | Default: 2 |
metricPrintPeriodMinutes | Duration in minutes between each set of metrics that are reported in the console Set the value to zero ( | Default: 5 |
metricLogLevel | Records the rate of logged events by their logging level Ensure that the following properties are set in the application.properties to enable logging:
| Default: Options:
|
Configuration for Flat File transformer
Configuration Properties | Description | Value |
|---|---|---|
| Required. The ID of the Gospel record definition, to which the data is imported. The ID field must begin with a letter ([A-Za-z]) |
Configuration for JSON transformer
| Configuration Properties | Value | |
|---|---|---|
pathToDefinition | Optional. Enter the location of the directory where the properties file is located. When importing JSON files, you have to map the properties in the file to the fields in the record. This mapping information is defined in the properties file. | Default: ./definitions.properties |
allowFileUploads | Optional. Set to true, to allow files to be uploaded into a Gospel record field | Default: false |
maxFileSizeMB | Optional. The maximum size (in MB) for a file to be uploaded into a Gospel record field | Default: 2 |
jsonTransformerQueueSize | Optional. The size of the transformer internal queue, which is the number of items that can be added to the queue. When the queue limit is reached, new records are blocked until the records are consumed. | Default: 1000 |
metricPrintPeriodMinutes | Duration in minutes between each set of metrics that are reported in the console Set the value to zero ( | Default: 5 |
metricLogLevel | Records the rate of logged events by their logging level Ensure that the following properties are set in the application.properties to enable logging:
| Default: Options:
|
Configuration for XML transformer
| Configuration Properties | Description | Value |
|---|---|---|
| Optional. Provide the location of the directory where the XSLT (eXtensible Stylesheet Language Transformations) file is located. If set, XSLT transformation is performed on the XML source documents before importing the records. | |
allowFileUploads | Optional. Set to true, to allow files to be uploaded into a Gospel record field | Default: false |
maxFileSizeMB | Optional. The maximum size (in MB) for a file to be uploaded into a Gospel record field | Default: 2 |
pathToDefinition | Required. Enter the location of the directory where the properties file is located. When importing XML files, you have to map the properties in the file to the fields in the record. This mapping information is defined in the properties file. | |
| Optional. Set to true, to enable internal DTD (Document Type Definition) verification for validating the document structure of the XML file being processed. For example: <?xml version="1.0" encoding="utf-8"?> <!--DTD is declared inside<!DOCTYPE> definition--> <!DOCTYPE Employee [ <!ELEMENT Employee (Details+)> <!ELEMENT Details (First_Name,Last_Name,Address)> <!ELEMENT First_Name (#PCDATA)> <!ELEMENT Last_Name (#PCDATA)> <!ELEMENT Address (#PCDATA)> ]> <Employee> <Details> <First_Name>John</First_Name> <Last_Name>Smith</Last_Name> <Address>London</Address> </Details> </Employee> | Default: false |
pathToXSDFile | Optional. XSD files are XML schemas that describe the structure of an XML document. If the validation fails, errors and warnings are reported in the console and the import may fail. For example, if your XML file contains data: <Employee> <Details> <First_Name>John</First_Name> <Last_Name>Smith</Last_Name> <Address>London</Address> </Details> </Employee> The XML schema (XSD) file can be written as: <?xml version="1.0" encoding="utf-8"?>
<xs:schema attributeFormDefault="unqualified" elementFormDefault="qualified" xmlns:xs="http://www.w3.org/2001/XMLSchema">
<xs:element name="Employee">
<xs:complexType mixed="true">
<xs:sequence>
<xs:element name="Details">
<xs:complexType mixed="true">
<xs:sequence>
<xs:element name="First_Name" type="xs:string" />
<xs:element name="Last_Name" type="xs:string" />
<xs:element name="Address" type="xs:string" />
</xs:sequence>
</xs:complexType>
</xs:element>
</xs:sequence>
</xs:complexType>
</xs:element>
</xs:schema>
| |
xmlTransformerQueueSize | Optional. The size of the transformer internal queue, which is the number of items that can be added to the queue. When the queue limit is reached, new records are blocked until the records are consumed. | Default: 1000 |
xsltBufferSize | Optional. Size of buffer for processing XSLT in bytes. | Default: Max: |
metricPrintPeriodMinutes | Duration in minutes between each set of metrics that are reported in the console Set the value to zero ( | Default: 5 |
metricLogLevel | Records the rate of logged events by their logging level Ensure that the following properties are set in the application.properties to enable logging:
| Default: Options:
|
Example Configuration
The application.properties file contains configurations that can be provided when starting the LedgerBridge Connector, with the required transformer and collector. Here are examples of the available parameters:
Main LedgerBridge connector configuration
Below listed are the parameters required for the LedgerBridge Connector:
httpRequestTimeoutSeconds=30 ledgerbridgeBaseUrl=http://localhost:8080 recordEndpoint=/api/post heartbeatEndpoint=/api/heartbeat configEndpoint=/api/config userAgent=GospelCollector preserveSecret=true sharedSecretFilename=.ledgerbridge qualifiedPathToSharedSecretDirectory=. secretLength=64 sharedSecretTokenValidityPeriodSeconds=15 batchSize=10 batchTimeoutSeconds=1 backOffQueueSize=100 metricPrintPeriodMinutes=2 metricLogLevel=DEBUG
Configuration for file collector
Below listed are the parameters required for importing data from a local source file. For example, CSV, JSON, XML, or FlatfiledataLocation=<directory-name> searchFilemask=.*\.<*.file-extension> moveAfterProcessing=false deleteAfterProcessing=false moveProcessedFilesTo=./processed renameProcessedFilesTo=.old metricPrintPeriodMinutes=2 metricLogLevel=DEBUG
Configuration for API collector
Below listed are the parameters required for importing data from an online source file.
url=https://source/file/location/example_1.json method=GET headers.AuthorizationToken=MWxheyJqdGkicGVuc2VzMW1l pollingTime=30 metricPrintPeriodMinutes=2 metricLogLevel=DEBUG
Configuration for CSV Collector Plugin
Below listed are the parameters required for importing data from a CSV file
pathToDefinition=./definitions.properties recordDefinitionId=<recorddefinition-id> customCsvDelimiter=, customCsvRegex=(?=(?:[^\"]*\"[^\"]*\")*[^\"]*$) customCsvHeaderLine=Name,ID,Department allowFileUploads=false maxFileSizeMB=2 metricPrintPeriodMinutes=2 metricLogLevel=DEBUG
Configuration for flatfile transformer
Below listed are the parameters required for importing data from a flatfile
recordDefinitionId=<recorddefinition-id> metricPrintPeriodMinutes=2 metricLogLevel=DEBUG
Configuration for JSON transformer
Below listed are the parameters required for importing data from a JSON file
pathToDefinition=./definitions.properties allowFileUploads=false maxFileSizeMB=2 jsonTransformerQueueSize=1000 metricPrintPeriodMinutes=2 metricLogLevel=DEBUG
Configuration for XML transformer
Below listed are the parameters required for importing data from an XML file
xsltLocation=<directory-name> allowFileUploads=false maxFileSizeMB=2 pathToDefinition=./definitions.properties dtdVerificationEnabled=false pathToXSDFile=<directory-name> xmlTransformerQueueSize=1000 useSaxonJar=false xsltBufferSize=1048576 metricPrintPeriodMinutes=2 metricLogLevel=DEBUG
Configuration for the JDBC collector plugin
Below listed are the parameters required for importing data from a JDBC database.
sqlQuery= jdbcUrl= jdbcUsername= jdbcPassword= pollingTime=10 fetchSize=10 metricPrintPeriodMinutes=2 metricLogLevel=DEBUG