Where change is welcome.
One of the most exciting new capabilities of StreamSets DataOps Platform is its ability to dynamically provision Public Cloud VMs running Data Collector or Transformer for Spark engines. Public cloud VMs running StreamSets engines can be deployed "just in time"…
This post describes how StreamSets Data Collector, a fast and flexible data ingestion engine, can be configured to take advantage of IAM Roles when running on AWS Elastic Kubernetes Service (EKS). IAM Roles allow data engineers to concentrate on business…
An increasing number of StreamSets' customers are deploying Data Collector Engine on Kubernetes, taking advantage of the reliability and elastic scaling Kubernetes provides. This post describes best practices for Kubernetes, configuring and managing the Data Collector Engine. More information and…
In addition to StreamSets Data Collector's support for a variety of Credential Stores, one can also use Kubernetes Secrets as a mechanism for securely managing credentials and environment-specific properties within Data Collector pipelines. This post shows an example of using…
A major challenge when deploying data pipelines to run on Kubernetes is how to handle Kerberos principals and Kerberos keytabs needed when pipelines write to secure Hadoop. What Are Kerberos Principals? Kerberos principals are identifiers that represent either users or…
Hari Nayak's recent blog post provides a quickstart for using StreamSets Control Hub to deploy multiple instances of StreamSets Data Collector on Google's Kubernetes Engine (GKE). This post modifies the core scripts from that project in order to run on Minikube rather than GKE. As Minikube can run…