StreamSets is excited to announce the immediate availability of StreamSets Data Collector 3.10.0 and StreamSets Data Collector Edge 3.10.0.
StreamSets Data Collector is open source under Apache License 2.0 and a powerful design and execution engine. It enables moving data between any source and destination, performing transformations, and push down analytics along the way. To download, click here.
StreamSets Data Collector Edge is a lightweight execution agent that runs on edge devices with limited memory, CPU, and/or connectivity resources. It enables reading data from an edge device or receiveing data from another dataflow pipeline. It supports messaging protocols including HTTP, MQTT, CoAP, and WebSockets. To download, click here.
Let’s review some of the highlights. For a complete list of enhancements, new features, bug fixes, and upgrade instructions, please refer to the Release Notes.
StreamSets Data Collector 3.10.0
This release includes the following new origins.
- Groovy Scripting—Runs a Groovy script to create Data Collector records.
- Jython Scripting—Runs a Jython script to create Data Collector records.
- NiFi HTTP Server—Listens for requests from a NiFi PutHTTP processor and processes NiFi FlowFiles.
- User-defined parameters—On the Advanced tab, enter parameters and values. In the script, access the value with the sdc.userParams dictionary.
- Full-screen script editing—With your cursor in the script field, press F11 or Esc on Mac OS X to toggle full-screen editing.
This release includes enhancements to the following destinations.
- Cassandra—The destination has four new properties to help you debug: Connection Timeout, Read Timeout, Consistency Level, and Log Slow Queries.
- RabbitMQ Producer—The destination has a new Set Expiration property, available when setting AMQP message properties on the RabbitMQ tab. Clear the Set Expiration property to disable expiration on messages that the destination sends.
This release includes the following enhancement.
JDBC Query—The executor can now run queries in parallel to improve throughput. On the Advanced tab, select the Enable Parallel Queries property to have the executor run queries simultaneously on each connection to the database.
Data Collector Configuration
This release includes the following Data Collector configuration enhancement.
The Data Collector configuration file sdc.properties contains a new property, stage.conf_com.streamsets.pipeline.stage.jdbc.drivers.load, where you can list JDBC drivers that Data Collector automatically loads for all pipelines.
Feedback and Contributions
If you’d like to suggest a feature, enhancement, or if you see something that needs to be fixed or made better, feel free to open a ticket by visiting—https://issues.streamsets.com.
Also note that StreamSets welcomes contributions from the community. For guidelines on contributing code, visit—https://github.com/streamsets/datacollector/blob/master/CONTRIBUTING.md
For any other questions and inquiries, please contact us.