Step 2- Configure and start the LedgerBridge Connector

In this section:

Prerequisites

Downloadable JAR files

The LedgerBridge connector supports the import of data from various types of files. Before you begin, download the LedgerBridge Connector JAR file and the required Connector-Collector and Connector-Transformer files from the Gospel DashboardDownloads section. 

Starting the LedgerBridge Connector

You can start LedgerBridge Connector either by defining the configuration in the application.properties file or by passing the parameters as command-line arguments.

Using application.properties

You can store configurable properties in an application.properties file and place it in the same directory as the JAR file.

Each parameter is stored as a pair of strings, one storing the name of the parameter, and the other storing the value. For example, dataLocation=datafiles where, dataLocation is the parameter and datafiles is the value. 

The parameters set from the command line have precedence over the properties file in the project directory.

If you have defined the configuration in the application.properties file, in the command prompt, go to the location of the directory where the gospel-ledgerbridge-<version>.jar file is located, and run the following command:

java -jar ledgerbridge-connector-<version>.jar

The LedgerBridge Administrative Panel > Connector Builder page displays the configuration currently applied when starting the LedgerBridge Connector. 

On this page, you can provide the required configuration and generate a configuration file using the Download Config option. Save the file to the LedgerBridge Connector directory as application.properties and restart the LedgerBridge Connector to apply the new configuration. 

Pass parameters in the terminal as command-line arguments

To pass the parameters in the command line arguments, go to the location of the directory where the gospel-ledgerbridge-<version>.jar file is located, and run the following command:

java -jar ledgerbridge-connector-<version>.jar --dataLocation=<path/to/directory> --ledgerbridgeBaseUrl=<http://localhost:port-number> 

The code example lists some configuration values required for starting the LedgerBridge Connector. You can add additional configuration values as described in the table below.

To see all the parameters you can list in the applicaiton.properties, execute:

java -jar ledgerbridge-connector-<version>.jar --show-params

To see only the mandatory parameters execute: 

java -jar ledgerbridge-connector-<version>.jar --show-mandatory-params

Connector Builder Settings 

The table lists and describes the configuration values you can set when starting the LedgerBridge Connector:

If you make changes to the configuration, you must restart the LedgerBridge Connector to apply the changes. 

Main LedgerBridge Connector configuration

Configuration PropertiesDescriptionValue
httpRequestTimeoutSecondsOptional. The timeout time in seconds for the HTTP client that connects to the LedegerBridgeDefault: 30
ledgerbridgeBaseUrlOptional. The LedgerBridge base URL in the form <protocol>://<host>:<port>Default: http://localhost:8090
recordEndpointOptional. The LedgerBridge record endpoint to which the LedgerBridge connector sends recordsDefault: /api/post

heartbeatEndpoint

Optional. The LedgerBridge heartbeat endpoint to which the LedgerBridge connector sends a periodical requestDefault: /api/heartbeat
configEndpointOptional. The LedgerBridge endpoint the to which the LedgerBridge connector sends its configurationDefault: /api/config
userAgentOptional. The User-Agent header sent by the HTTP client to the LedgerBridgeDefault: GospelCollector
preserveSecretOptional. When set to true, the LedgerBridge Connector attempts to load secret from the file system, based on the parameters passed. Otherwise, it generates a random secret.Default: true
sharedSecretFilenameOptional. The file containing the secret passphrase that is shared between Ledgerbridge and Ledgerbridge connectorDefault: .ledgerbridge
qualifiedPathToSharedSecretDirectory

Optional. The directory containing the secret passphrase that is shared between Ledgerbridge and Ledgerbridge connectorDefault: .

secretLength

Optional. The shared secret length in bytesDefault: 64
sharedSecretTokenValidityPeriodSecondsOptional. The period (in seconds) after which a LedgerBridge connection-token becomes invalid Default: 15
batchSize

Optional. The records are sent to the LedgerBridge in batches. Set the number of records to be sent in each batch.

For example, if you set the value to a big number, more data is sent to the LedgerBridge, and if the value is set to a small number, frequent requests are made to the LedgerBridge for data.

Default: 10
backOffQueueSize

Optional. The size of the internal backoff queue. If an error is reported when processing data, the records are added to the backOff queue until the error is resolved. 

Default: 100
batchTimeoutSecondsOptional. The timeout after which a record batch gets consumed, even if it has not reached the batch size.Default value: 1
metricPrintPeriodMinutes

Duration in minutes between each set of metrics that are reported in the console

Set the value to zero (0) to disable the feature.

Default: 5
metricLogLevel

Records the rate of logged events by their logging level

Ensure that the following properties are set in the application.properties to enable logging:

logging.level.com.gospel=DEBUG

logging.level.tech.gospel=DEBUG

Default: DEBUG

Options: 

  • DEBUG
  • ERROR
  • INFO
  • TRACE
  • WARN 

Configuration for File Collector

Configuration Properties

DescriptionValue
dataLocation

Required. The directory where the source files are located. You can change the default (root) location by providing the path to the folder containing the source data files. 

Default: .

For example,  /Users/jsmith/sourcefiles

searchFilemask

Required. Set the file mask to specify the files to be processed and the files to be ignored. 

To import data from compressed files, enter the extension of the file. For example, .tar.gz, .tar, tar.bz2, or .zip.

Default: .*

For example, .*\.xml

moveAfterProcessingOptional. If set to true, the processed files are moved to a sub-directory.Default: false
moveProcessedFilesTo

Optional. If the moveAfterProcessing property is set to true, select the location where you want to save the processed files.

Default: ./processed
deleteAfterProcessingOptional. If set to true, all the processed files are deleted. Default: false
renameProcessedFilesToOptional. By default, all processed files are renamed. Specify the suffix to append to the processed file name.Default: .old
charset

Specifies the character encoding used in the files to be imported.

If the encoding value of the file is different from the encoding value of the system, the import may fail.  

For example: UTF-8
metricPrintPeriodMinutes

Duration in minutes between each set of metrics that are reported in the console

Set the value to zero (0) to disable the feature.

Default: 5
metricLogLevel

Records the rate of logged events by their logging level

Ensure that the following properties are set in the application.properties to enable logging:

logging.level.com.gospel=DEBUG

logging.level.tech.gospel=DEBUG

Default: DEBUG

Options: 

  • DEBUG
  • ERROR
  • INFO
  • TRACE
  • WARN 

Configuration for API collector

Configuration Properties

DescriptionValue
urlRequired. The URL to the source file.
methodRequired. The API method to use. 

Options: 

  • GET 
  • POST

headers

Optional. The HTTP headers to pass to the API invocation.

Configure as:

headers.<headerName>=<headerValue>

pollingTimeOptional. The pause (in seconds) between each API callDefault: 30
metricPrintPeriodMinutes

Duration in minutes between each set of metrics that are reported in the console

Set the value to zero (0) to disable the feature.

Default: 5
metricLogLevel

Records the rate of logged events by their logging level

Ensure that the following properties are set in the application.properties to enable logging:

logging.level.com.gospel=DEBUG

logging.level.tech.gospel=DEBUG

Default: DEBUG

Options: 

  • DEBUG
  • ERROR
  • INFO
  • TRACE
  • WARN 

Configuration for the JDBC collector plugin

Configuration PropertiesDescriptionValue

sqlQuery

Required. The SQL query to execute on the database to which the plugin is connected
jdbcUrlRequired. The URL of the JDBC database to which the plugin connects
jdbcUsernameRequired. The database username used to connect to the database
jdbcPasswordOptional. The password of the database user (if required) 
pollingTimeOptional. The interval (in seconds) between each SQL query that is executed against the databaseDefault: 10
fetchSizeOptional. Determines how many records to fetch at a time from the database for processingDefault: 10
metricPrintPeriodMinutes

Duration in minutes between each set of metrics that are reported in the console

Set the value to zero (0) to disable the feature.

Default: 5
metricLogLevel

Records the rate of logged events by their logging level

Ensure that the following properties are set in the application.properties to enable logging:

logging.level.com.gospel=DEBUG

logging.level.tech.gospel=DEBUG

Default: DEBUG

Options: 

  • DEBUG
  • ERROR
  • INFO
  • TRACE
  • WARN 

Configuration for CSV transformer

Configuration PropertiesDescriptionValue
pathToDefinition

Optional. Enter the location of the directory where the properties file is located. By default, the properties file must be placed in the directory where the LedgerBridge connector, collector and transformer JAR files are located.

When importing CSV files, the headers in the CSV file are automatically mapped to fields in the record and therefore does not require a definitions file. You can, however, provide mapping using the definitions.properties file or you can define the headers in the application.properties file, in the parameter. 

Default: ./definitions.properties
recordDefinitionId

Optional. The ID of the Gospel record definition, to which the data is imported

The ID field must begin with a letter ([A-Za-z])


customCsvDelimiterOptional. Set a specific column delimiter. During import, the delimiter helps identify where each field begins and ends in the imported file. Default: ,
customCsvRegexOptional. Specify a custom behaviour when splitting columns. For example, you may want to consider the string "abc, def" (enclosed within double quotes) as a single column.Default: (?=(?:[^\"]*\"[^\"]*\")*[^\"]*$)
customCsvHeaderLine

Optional. Specify a custom header for the CSV files being imported. This is useful when the input files do not have a header line.

For example: customCsvHeaderLine=field1,field2,field3


allowFileUploads Optional. Set to true, to allow files to be uploaded into a Gospel record fieldDefault: false
maxFileSizeMBOptional. The maximum size (in MB) for a file to be uploaded to a field in a recordDefault: 2
metricPrintPeriodMinutes

Duration in minutes between each set of metrics that are reported in the console

Set the value to zero (0) to disable the feature.

Default: 5
metricLogLevel

Records the rate of logged events by their logging level

Ensure that the following properties are set in the application.properties to enable logging:

logging.level.com.gospel=DEBUG

logging.level.tech.gospel=DEBUG

Default: DEBUG

Options: 

  • DEBUG
  • ERROR
  • INFO
  • TRACE
  • WARN 

Configuration for Flat File transformer

Configuration Properties

DescriptionValue

recordDefinitionId

Required. The ID of the Gospel record definition, to which the data is imported.

The ID field must begin with a letter ([A-Za-z])


Configuration for JSON transformer

Configuration Properties
Value
pathToDefinitionOptional. Enter the location of the directory where the properties file is located. 

When importing JSON files, you have to map the properties in the file to the fields in the record. This mapping information is defined in the properties file.

Default: ./definitions.properties
allowFileUploadsOptional. Set to true, to allow files to be uploaded into a Gospel record fieldDefault: false
maxFileSizeMBOptional. The maximum size (in MB) for a file to be uploaded into a Gospel record fieldDefault: 2
jsonTransformerQueueSizeOptional. The size of the transformer internal queue, which is the number of items that can be added to the queue. When the queue limit is reached, new records are blocked until the records are consumed.Default: 1000
metricPrintPeriodMinutes

Duration in minutes between each set of metrics that are reported in the console

Set the value to zero (0) to disable the feature.

Default: 5
metricLogLevel

Records the rate of logged events by their logging level

Ensure that the following properties are set in the application.properties to enable logging:

logging.level.com.gospel=DEBUG

logging.level.tech.gospel=DEBUG

Default: DEBUG

Options: 

  • DEBUG
  • ERROR
  • INFO
  • TRACE
  • WARN 

Configuration for XML transformer

Configuration PropertiesDescriptionValue

xsltLocation

Optional. Provide the location of the directory where the XSLT (eXtensible Stylesheet Language Transformations) file is located. If set, XSLT transformation is performed on the XML source documents before importing the records.


allowFileUploadsOptional. Set to true, to allow files to be uploaded into a Gospel record fieldDefault: false
maxFileSizeMBOptional. The maximum size (in MB) for a file to be uploaded into a Gospel record fieldDefault: 2
pathToDefinitionRequired. Enter the location of the directory where the properties file is located. 

When importing XML files, you have to map the properties in the file to the fields in the record. This mapping information is defined in the properties file.


dtdVerificationEnabled

Optional. Set to true, to enable internal DTD (Document Type Definition) verification for validating the document structure of the XML file being processed.

For example:

<?xml version="1.0" encoding="utf-8"?> 
<!--DTD is declared inside<!DOCTYPE> definition-->
<!DOCTYPE Employee [
<!ELEMENT Employee (Details+)>
<!ELEMENT Details (First_Name,Last_Name,Address)>
        <!ELEMENT First_Name (#PCDATA)>
        <!ELEMENT Last_Name (#PCDATA)>
        <!ELEMENT Address (#PCDATA)>
]>
<Employee> 
  <Details> 
      <First_Name>John</First_Name> 
      <Last_Name>Smith</Last_Name> 
      <Address>London</Address> 
    </Details> 
</Employee>
Default: false
pathToXSDFile

Optional. XSD files are XML schemas that describe the structure of an XML document. If the validation fails, errors and warnings are reported in the console and the import may fail.

For example, if your XML file contains data:

<Employee> 
  <Details> 
      <First_Name>John</First_Name> 
      <Last_Name>Smith</Last_Name> 
      <Address>London</Address> 
    </Details> 
</Employee>

The XML schema (XSD) file can be written as:

<?xml version="1.0" encoding="utf-8"?>
<xs:schema attributeFormDefault="unqualified" elementFormDefault="qualified" xmlns:xs="http://www.w3.org/2001/XMLSchema">
  <xs:element name="Employee">
    <xs:complexType mixed="true">
      <xs:sequence>
        <xs:element name="Details">
          <xs:complexType mixed="true">
            <xs:sequence>
              <xs:element name="First_Name" type="xs:string" />
              <xs:element name="Last_Name" type="xs:string" />
              <xs:element name="Address" type="xs:string" />
            </xs:sequence>
          </xs:complexType>
        </xs:element>
      </xs:sequence>
    </xs:complexType>
  </xs:element>
</xs:schema>

xmlTransformerQueueSizeOptional. The size of the transformer internal queue, which is the number of items that can be added to the queue. When the queue limit is reached, new records are blocked until the records are consumed.Default: 1000
xsltBufferSizeOptional. Size of buffer for processing XSLT in bytes.

Default: 1MB (1048576)

Max: 1GB

metricPrintPeriodMinutes

Duration in minutes between each set of metrics that are reported in the console

Set the value to zero (0) to disable the feature.

Default: 5
metricLogLevel

Records the rate of logged events by their logging level

Ensure that the following properties are set in the application.properties to enable logging:

logging.level.com.gospel=DEBUG

logging.level.tech.gospel=DEBUG

Default: DEBUG

Options: 

  • DEBUG
  • ERROR
  • INFO
  • TRACE
  • WARN

Example Configuration

The application.properties file contains configurations that can be provided when starting the LedgerBridge Connector, with the required transformer and collector. Here are examples of the available parameters:

Main LedgerBridge connector configuration

Below listed are the parameters required for the LedgerBridge Connector: 

httpRequestTimeoutSeconds=30
ledgerbridgeBaseUrl=http://localhost:8080
recordEndpoint=/api/post
heartbeatEndpoint=/api/heartbeat
configEndpoint=/api/config
userAgent=GospelCollector
preserveSecret=true
sharedSecretFilename=.ledgerbridge
qualifiedPathToSharedSecretDirectory=.
secretLength=64
sharedSecretTokenValidityPeriodSeconds=15
batchSize=10
batchTimeoutSeconds=1
backOffQueueSize=100
metricPrintPeriodMinutes=2
metricLogLevel=DEBUG

Configuration for file collector 

Below listed are the parameters required for importing data from a local source file. For example, CSV, JSON, XML, or Flatfile
dataLocation=<directory-name>
searchFilemask=.*\.<*.file-extension>
moveAfterProcessing=false
deleteAfterProcessing=false
moveProcessedFilesTo=./processed
renameProcessedFilesTo=.old
metricPrintPeriodMinutes=2
metricLogLevel=DEBUG

Configuration for API collector 

Below listed are the parameters required for importing data from an online source file.

url=https://source/file/location/example_1.json
method=GET
headers.AuthorizationToken=MWxheyJqdGkicGVuc2VzMW1l 
pollingTime=30
metricPrintPeriodMinutes=2
metricLogLevel=DEBUG

Configuration for CSV Collector Plugin

Below listed are the parameters required for importing data from a CSV file

pathToDefinition=./definitions.properties
recordDefinitionId=<recorddefinition-id>
customCsvDelimiter=,
customCsvRegex=(?=(?:[^\"]*\"[^\"]*\")*[^\"]*$)
customCsvHeaderLine=Name,ID,Department
allowFileUploads=false
maxFileSizeMB=2
metricPrintPeriodMinutes=2
metricLogLevel=DEBUG

Configuration for flatfile transformer

Below listed are the parameters required for importing data from a flatfile

recordDefinitionId=<recorddefinition-id>
metricPrintPeriodMinutes=2
metricLogLevel=DEBUG

Configuration for JSON transformer

Below listed are the parameters required for importing data from a JSON file

pathToDefinition=./definitions.properties
allowFileUploads=false
maxFileSizeMB=2
jsonTransformerQueueSize=1000
metricPrintPeriodMinutes=2
metricLogLevel=DEBUG

Configuration for XML transformer

Below listed are the parameters required for importing data from an XML file

xsltLocation=<directory-name>
allowFileUploads=false
maxFileSizeMB=2
pathToDefinition=./definitions.properties
dtdVerificationEnabled=false
pathToXSDFile=<directory-name>
xmlTransformerQueueSize=1000
useSaxonJar=false
xsltBufferSize=1048576
metricPrintPeriodMinutes=2
metricLogLevel=DEBUG

Configuration for the JDBC collector plugin

Below listed are the parameters required for importing data from a JDBC database.

sqlQuery=
jdbcUrl=
jdbcUsername=
jdbcPassword=
pollingTime=10
fetchSize=10
metricPrintPeriodMinutes=2
metricLogLevel=DEBUG