Here are some of the new features and enhancements in 22.214.171.124. For a full list, see What's New. For a list of bug fixes and known issues, see the Release Notes.
Multithreaded Pipelines – The multithreaded framework includes the following enhancements:
- Origins for multithreaded pipelines – You can now use the following origins to create multithreaded pipelines:
– Elasticsearch origin
– JDBC Multitable Consumer origin
– Kinesis Consumer origin
– WebSocket Server origin
- Maximum pipeline runners – You can now configure a maximum number of pipeline runners to use in a pipeline. Previously, Data Collector generated pipeline runners based on the number of threads created by the origin. This allows you to tune performance and resource usage. By default, Data Collector still generates runners based on the number of threads that the origin uses.
- Record Deduplicator processor enhancement – The processor can now deduplicate records across all pipeline runners in a multithreaded pipeline.
- Pipeline validation enhancement – The pipeline now displays duplicate errors generated by using multiple threads as one error message.
- Log enhancement – Multithreaded pipelines now include the runner ID in log information.
- Monitoring – Monitoring now displays a histogram of available pipeline runners, replacing the information previously included in the Runtime Statistics list.
Stop pipeline execution – You can configure pipelines to transfer data and automatically stop execution based on an event such as reaching the end of a table. The JDBC Multitable Consumer, JDBC Query Consumer, and Salesforce origins can generate events when they reach the end of available data that the new Pipeline Finisher executor uses to stop the pipeline. Click here for a case study.
Pipeline Runtime Parameters – You can define runtime parameters when you configure a pipeline, and then call the parameters from within that pipeline. When you start the pipeline from the user interface, the command line, or the REST API, you specify the values to use for those parameters. Use pipeline parameters to represent any stage or pipeline property with a value that must change for each pipeline run – such as batch sizes and timeouts, directories, or URI.