Today we are pleased to announce the general availability of StreamSets Control Hub version 3.2. StreamSets has built the industry’s only DataOps platform. We call it DataOps because our platform makes it easy to iteratively update dataflows when technology changes. Release 3.2 enhances manageability and repeatability between Control Hub instances which is critical when understanding data operations across systems and data sources. The newest release also includes functionality that allows for quicker deployment of standardized data pipelines and faster, more reliable job scheduling. As data volumes grow and the business begins to consume data for a variety of analytics it’s increasingly important that data pipelines can be deployed, managed, and monitored comprehensively. Let’s take a look at some of the improvements.
Often times developers need to perform a common set of transformations across many pipelines. Replicating these efforts manually can be cumbersome and not the best use of your time. Fragments lets you maximize and reuse common transformations allowing for quick implementation of data pipelines. This feature is only available in Control Hub, and it is a means for development teams to create standards for data flow transformations and optimize their design efforts.
Data Delivery Reports
Users can view and publish reports including Data Delivery statistics for data flowing through pipelines. Their operations counterparts can use these reports as metrics for proving value and issuing chargebacks.
Move Control Hub Artifacts Between Instances
Often times operations needs to replicate a data pipeline experience across multiple environments like dev/test/production. You can now easily migrate artifacts such as Jobs and Pipelines between instances of Control Hub via a user interface. API’s can be leveraged for further automation.
Designed for batch workloads, Jobs can now be scheduled from within Control Hub. Users who choose not to use their enterprise scheduler can use this feature as an easy scheduler for StreamSets jobs. A combination of the scheduler and the Control Hub events mechanisms allows users to create basic workflows. Users can choose to leverage their current enterprise scheduler via API hooks or use the included Job scheduler for batch jobs.
Data professionals looking to evaluate Control Hub/DPM on-premises can now install a time bound trial of the software. The UI lets a user know how many days remain on the trial and provides options for continued usage.
Notifications and Alerts via Webhooks
Pipeline and job lifecycle events can now be published via Webhooks. StreamSets users can use this feature to ‘tune in’ to important events within Control Hub and consume them via an escalation/notification tool commonly used in their organization (e.g. Slack, PagerDuty, Zendesk). Developers can be notified if any of their team members update a given pipeline and operations can be notified if a job suddenly stops functioning.
StreamSets Control Hub version 3.2 also includes improvements for easy platform installation, seamless scaling of Kubernetes resources, and removed dependency of the included time-series database.