Where Change Is Welcome
An increasing number of StreamSets' customers are deploying Data Collector on Kubernetes, taking advantage of the reliability and elastic scaling Kubernetes provides. This post describes best practices for configuring and managing Data Collector on Kubernetes. More information and in-depth examples…
In this blog, we will review how easy it is to set up an end-to-end ETL data pipeline that runs on StreamSets Transformer to perform extract, transform, and load (ETL) operations. The pipeline uses Apache Spark for Azure HDInsight cluster…
StreamSets is proud to announce an expansion of its partnership with Databricks by participating in Databricks’ newly launched Data Ingestion Network. A key component of this integration is leveraging the recently released Databricks COPY command for bulk ingest into Databricks…
In this post, we will explore how to bulk ingest and process change data capture (CDC) information from Oracle 19c database using the enhanced Oracle CDC Client origin into Databricks Delta Lake using StreamSets Data Collector. You’ll also learn one…
StreamSets is excited to announce the immediate availability of StreamSets Data Collector 3.16.0. StreamSets Data Collector is a powerful design and execution engine. It enables you to build and deploy intent driven and data drift resilient smart data pipelines. Highlights…
In addition to StreamSets Data Collector's support for a variety of Credential Stores, one can also use Kubernetes Secrets as a mechanism for securely managing credentials and environment-specific properties within Data Collector pipelines. This post shows an example of using…
StreamSets Data Collector’s HTTP pipeline stages allow a wide range of API integrations. I recently built a data pipeline to read customer data from a MySQL database: retrieving, creating and updating leads in Marketo via their REST API. I’ll share…
StreamSets Data Collector is available in the major cloud marketplaces; as an AWS Marketplace vendor, StreamSets has access to a number of data sets detailing product usage and billing. Data sets are available interactively via a web portal and via…
Learn how quickly you can start ingesting and aggregating clickstream logs using StreamSets Transformer running on Amazon EMR, and see how the data is analyzed in Elasticsearch, Kibana, and Amazon Redshift. Clickstream analysis by definition is the process of collecting,…
Learn how to load a serialized Spark ML model stored in MLeap bundle format on Databricks File System (DBFS), and use it for classification on new, streaming data flowing through the StreamSets DataOps Platform. In my previous blogs, I illustrated…