Kinesis Producer
The Kinesis Producer destination writes data to Amazon Kinesis Streams. The destination can also send responses to a microservice origin when used in a microservice pipeline.
To write data to an Amazon Kinesis Firehose delivery system, use the Kinesis Firehose destination. To write data to Amazon S3, use the Amazon S3 destination.
When you configure Kinesis Producer, you specify the Amazon Web Services connection information for your Kinesis cluster and the data format to use. When you want the destination to send responses to a microservice origin within a microservice pipeline, you specify the type of response to send.
- Writes a maximum of 500 messages and 4.5 MB in a batch.
- Allows a maximum of 50 kb/message. Larger messages are sent to the stage for error handling.
AWS Credentials
When Data Collector writes data to a Kinesis Producer destination, it must pass credentials to Amazon Web Services.
Use one of the following methods to pass AWS credentials:
- IAM roles
- When Data Collector runs on an Amazon EC2 instance, you can use the AWS Management Console to configure an IAM role for the EC2 instance. Data Collector uses the IAM instance profile credentials to automatically connect to AWS.
- AWS access key pairs
-
When Data Collector does not run on an Amazon EC2 instance or when the EC2 instance doesn’t have an IAM role, you must specify the Access Key ID and Secret Access Key properties in the destination.
Send Microservice Responses
The Kinesis Producer destination can send responses to a microservice origin when you use the destination in a microservice pipeline.
- All successfully written records.
- Responses from the destination system - For information about the possible responses, see the documentation for the destination system.
Data Formats
Kinesis Producer writes data to Kinesis based on the data format that you select. You can use the following data formats:
- Avro
- The destination writes records based on the Avro schema. You can use one of the following methods to specify the location of the Avro schema definition:
- Binary
- The destination writes binary data from a single field in the record.
- Delimited
- The destination writes records as delimited data. When you use this data format, the root field must be list or list-map.
- JSON
- The destination writes records as JSON data. You can use one of
the following formats:
- Array - Each file includes a single array. In the array, each element is a JSON representation of each record.
- Multiple objects - Each file includes multiple JSON objects. Each object is a JSON representation of a record.
- Protobuf
- Writes one record in a message. Uses the user-defined message type and the definition of the message type in the descriptor file to generate the message.
- SDC Record
- The destination writes records in the SDC Record data format.
- Text
- The destination writes data from a single text field to the destination system. When you configure the stage, you select the field to use. When necessary, merge record data into the field earlier in the pipeline.
Configuring a Kinesis Producer Destination
Configure a Kinesis Producer destination to write data to Amazon Kinesis Streams.