Disclaimer

Your use of this download is governed by Stonebranch’s Terms of Use, which are available at https://www.stonebranch.com/integration-hub/Terms-and-Privacy/Terms-of-Use/

Overview

Apache Kafka is an open-source distributed event streaming platform used by thousands of companies for high-performance data pipelines, streaming analytics, data integration, and mission-critical applications. This Universal Extension is responsible for publishing events (messages) to topics in Kafka.

Software Requirements

This integration requires a Universal Agent and a Python runtime to execute the Universal Task.

Software Requirements for Universal Template and Universal Task

Requires Python 3.7.0 or higher. Tested with the Universal Agent bundled Python distribution.

Software Requirements for Universal Agent

Both Windows and Linux agents are supported:

• Universal Agent for Windows x64 Version 7.0.0.0 and later.

• Universal Agent for Linux Version 7.0.0.0 and later.

Software Requirements for Universal Controller

Universal Controller Version 7.0.0.0 and later.

Software Requirements in Cases of SSL Verification

• If Python 3.7 Distribution for Universal Agent is installed, the path location of the Certificate Authority file must be provided.

• If Python is installed on the local system but not with the Universal Agent Distribution:

  • If the SSL certificate is signed by a publicly trusted certificate authority (CA), it is not required to provide the certificate.

  • If the SSL certificate is self-signed, either the path location of the certificate file must be provided, or the certificate must be stored at the machine's default location for installed certificates (in this case it will be trusted for all applications).

Supported Apache Kafka versions

This Universal Extension supports the Apache Kafka versions 2.0.0 and above. Because the Kafka server protocol is backwards compatible, the extension is expected to work with versions 0.8.0 and above (except 0.8.2-beta release); however, this has not been tested.

Key Features

This Universal Extension supports the following main features:

• Perform authentication towards Kafka, using PLAINTEXT or SASL_SSL SCRAM security protocol.

• Send a message to Kafka with the capability to select, the topic, the partition, the message key, the message value, and message metadata.

• Capability to control the transport of the messages by configuring the message acknowledgment strategy and the request timeout.

• Capability to fetch topics and partitions dynamically from Kafka for selection during task creation.

• Capability to automatically select the serialization method depending on the key/value message data types.

Import the Universal Template

To use this downloadable Universal Template, you first must perform the following steps:

1. This Universal Task requires the Resolvable Credentials feature. Check that the Resolvable Credentials Permitted system property has been set to true.

2. Download the provided ZIP file.

3. In the Universal Controller UI, select Administration >Configuration > Universal Templates to display the current list of Universal Templates.

4. Click Import Template.

5. Select the template ZIP file and Import.

When the template has been imported successfully, the Universal Template will appear on the list. Refresh your Navigation Tree to see these tasks in the Automation Center Menu.

Configure Universal Task

For the new Universal Task type, create a new task, and enter the task-specific details that were created in the Universal Template.

Input Fields

The input fields for this Universal Extension are described below.

Task Output

Exit Codes

The exit codes for this Universal Extension are described below.

Task Examples

1. Give the task a name.

2. Select an active agent of version 7.0.0.0 or higher for the task to run on.

3. Enter the Bootstrap Server(s) separated by comma.

4. Click the Topic field search icon to display the Topic names list. Clicking the Topic search icon will open the following dialog.
   
   
   
   Click the Submit button and select your Topic from the drop-down list.

5. Click the Partition field search icon to bring the Partition names list. Clicking the Partition search icon will open the following dialog.
   
   
   
   

   Click the Submit button and select your Partition key from the drop-down list.

6. Select Acks if a value other than the default one is required.

7. Enter your message in YAML format in Message Payload field.
   
   Example of message with an integer as a key and an object as a value:
   
   
   
   
   

8. Add key value pairs to the Headers field if necessary.

9. Manually enter an integer value for Request Timeout field, if the default one is not the required one.

Message without Value

Example of message without value.

Message Payload Serialization

According to the key and value datatypes of the YAML message, the extension will handle their respective serialization before sending them to Kafka.

Examples of message payloads with different datatypes for key/value and how they are serialized:

Integer as a Key and an Integer as a Value

Both key and value will be serialized as integers:

YAML:

key: 1
value: 256

String as a Key and an Integer as a Value

Key will be serialized as a string (an integer in single or double quotes will be interpreted as string) and value will be serialized as an integer:

YAML:

key: "1"
value: 256

Integer as a Key and a Parameter as a Value

The value will be serialized according to the variable datatype:

YAML:

key: "1"
value: 256

Integer as a Key and an Object as a Value

The value which in this case is an object will be serialized as a JSON string:

YAML:

key: 1
value:
  employee:
    position: developer
    details:
     name: John
     surname: Smith
     age: 30

Document References

This document references the following documents: