Using StreamSets Control Hub for Scalable Deployment via Kubernetes
In Scaling out StreamSets with Kubernetes, I explained how to spin up Data Collectors as Kubernetes deployments along with Dataflow Performance Manager. I recommended using a deployment with one replica as the design environment and a deployment with many replicas for execution. We recently announced StreamSets Control Hub which makes the Kubernetes integration way smoother! StreamSets Control Hub adds a Control Agent for Kubernetes that supports creating and managing Data Collector deployments and a Pipeline Designer that allows designing pipelines for Kubernetes without having to install Data Collectors. In this blog, I will demonstrate how to take advantage of these features.