The DataOps Blog
Where Change Is Welcome
StreamSets Live: Demos with Dash!
45 minutes. Live demos. No (0) slides! Key takeaways: Visually build data pipelines like you code: build, verify, iterate, commit, rinse, repeat! Ingest data from…
Best Practices For Running StreamSets Data Collector On Kubernetes
An increasing number of StreamSets' customers are deploying Data Collector on Kubernetes, taking advantage of the reliability and elastic scaling Kubernetes provides. This post describes best practices for configuring and managing Data Collector on Kubernetes. More information and in-depth examples of the techniques below can be found in StreamSets' Kubernetes Deployment Tutorial. 1) Use Control Hub to manage Data Collector…
Spark ETL To Derive Sales Insights on Azure HDInsight And Power BI
In this blog, we will review how easy it is to set up an end-to-end ETL data pipeline that runs on StreamSets Transformer to perform extract, transform, and load (ETL) operations. The pipeline uses Apache Spark for Azure HDInsight cluster to extract raw data and transform it (cleanse and curate) before storing it in multiple destinations for efficient downstream analysis.…
Bulk Ingest of Salesforce Data Into Databricks Delta Lake
StreamSets is proud to announce an expansion of its partnership with Databricks by participating in Databricks’ newly launched Data Ingestion Network. A key component of this integration is leveraging the recently released Databricks COPY command for bulk ingest into Databricks Delta Lake using StreamSets for Databricks. Let’s consider a simple example of ingesting accounts information from Salesforce and storing it…
Oracle 19c Bulk Ingest And Change Data Capture Into Databricks Delta Lake
In this post, we will explore how to bulk ingest and process change data capture (CDC) information from Oracle 19c database using the enhanced Oracle CDC Client origin into Databricks Delta Lake using StreamSets Data Collector. You’ll also learn one way to automate and orchestrate the two jobs using StreamSets Control Hub REST APIs. Introduction With Databricks Runtime version 6.3…
Announcing StreamSets Data Collector 3.16.0
StreamSets is excited to announce the immediate availability of StreamSets Data Collector 3.16.0. StreamSets Data Collector is a powerful design and execution engine. It enables you to build and deploy intent driven and data drift resilient smart data pipelines. Highlights There are some great new features and enhancements included in this release—below I’ve reviewed some of the highlights. For a…