Where Change Is Welcome
This post describes how StreamSets Data Collector, a fast and flexible data ingestion engine, can be configured to take advantage of IAM Roles when running on AWS Elastic Kubernetes Service (EKS). IAM Roles allow data engineers to concentrate on business…
An increasing number of StreamSets' customers are deploying Data Collector on Kubernetes, taking advantage of the reliability and elastic scaling Kubernetes provides. This post describes best practices for configuring and managing Data Collector on Kubernetes. More information and in-depth examples…
In addition to StreamSets Data Collector's support for a variety of Credential Stores, one can also use Kubernetes Secrets as a mechanism for securely managing credentials and environment-specific properties within Data Collector pipelines. This post shows an example of using…
A major challenge when deploying dataflow pipelines to run on Kubernetes is how to handle Kerberos principals and keytabs needed when pipelines write to secure Hadoop. One approach, of using Kerberos keytabs for principals of the form @ (without a…
Hari Nayak's recent blog post provides a quickstart for using StreamSets Control Hub to deploy multiple instances of StreamSets Data Collector on Google's Kubernetes Engine (GKE). This post modifies the core scripts from that project in order to run on Minikube rather than GKE. As Minikube can run…