Pulsar Producer
Supported pipeline types:
|
When you configure a Pulsar Producer destination, you define the URL to connect to Pulsar. You also define the topics to publish messages to. You can configure the destination to publish messages to a single topic or to multiple topics if you include an expression in the configured topic name.
You can configure the destination to use Pulsar security features. You can also configure advanced properties as needed, such as the partition or compression type to use when publishing messages or whether the destination publishes messages asynchronously or synchronously.
For more information about Pulsar topics and producers, see the Apache Pulsar documentation.
Enabling Security
If the Pulsar cluster uses security features, you must configure the Pulsar Producer destination to use the same security features to connect to Pulsar.
A Pulsar cluster can use the following security features:
- TLS transport encryption
- When configured for TLS transport encryption, the Pulsar cluster uses TLS to encrypt all traffic between the Pulsar server and clients. The Pulsar server uses a key and certificate which clients use to verify the server's identity.
- Mutual TLS authentication
- When configured for TLS transport encryption, the Pulsar cluster can additionally be configured to use mutual TLS authentication. With mutual authentication, clients also use keys and certificates which the server uses to verify the client's identity.
Data Formats
The Pulsar Producer destination writes data to Pulsar based on the data format that you select. You can use the following data formats:
- Binary
- The stage writes binary data to a single field in the record.
- Delimited
- The destination writes records as delimited data. When you use this data format, the root field must be list or list-map.
- JSON
- The destination writes records as JSON data. You can use one of
the following formats:
- Array - Each file includes a single array. In the array, each element is a JSON representation of each record.
- Multiple objects - Each file includes multiple JSON objects. Each object is a JSON representation of a record.
- Protobuf
- Writes one record in a message. Uses the user-defined message type and the definition of the message type in the descriptor file to generate the message.
- SDC Record
- The destination writes records in the SDC Record data format.
- Text
- The destination writes data from a single text field to the destination system. When you configure the stage, you select the field to use.
- XML
- The destination creates a valid XML document for each record. The
destination requires the record to have a single root field that
contains the rest of the record data. For details and
suggestions for how to accomplish this, see Record Structure Requirement.
The destination can include indentation to produce human-readable documents. It can also validate that the generated XML conforms to the specified schema definition. Records with invalid schemas are handled based on the error handling configured for the destination.
Configuring a Pulsar Producer
Configure a Pulsar Producer destination to write data to Pulsar topics.