The DataOps Blog
Where Change Is Welcome
StreamSets Live: Demos with Dash!
45 minutes. Live demos. No (0) slides! Key takeaways: Visually build data pipelines like you code: build, verify, iterate, commit, rinse, repeat! Ingest data from…
Bulk Ingest of Salesforce Data Into Databricks Delta Lake
StreamSets is proud to announce an expansion of its partnership with Databricks by participating in Databricks’ newly launched Data Ingestion Network. A key component of this integration is leveraging the recently released Databricks COPY command for bulk ingest into Databricks Delta Lake using StreamSets for Databricks. Let’s consider a simple example of ingesting accounts information from Salesforce and storing it…
Oracle 19c Bulk Ingest And Change Data Capture Into Databricks Delta Lake
In this post, we will explore how to bulk ingest and process change data capture (CDC) information from Oracle 19c database using the enhanced Oracle CDC Client origin into Databricks Delta Lake in StreamSets Data Collector, a fast data ingestion engine. You’ll also learn one way to automate and orchestrate the two jobs using StreamSets Control Hub REST APIs. Introduction…
Announcing StreamSets Data Collector 3.16.0
StreamSets is excited to announce the immediate availability of StreamSets Data Collector 3.16.0. StreamSets Data Collector is a powerful design and execution engine. It enables you to build and deploy intent driven and data drift resilient smart data pipelines. Highlights There are some great new features and enhancements included in this release—below I’ve reviewed some of the highlights. For a…
Using Kubernetes Secrets in Data Collector Pipelines
In addition to StreamSets Data Collector's support for a variety of Credential Stores, one can also use Kubernetes Secrets as a mechanism for securely managing credentials and environment-specific properties within Data Collector pipelines. This post shows an example of using a Kubernetes Secret to manage MySQL connection properties for a JDBC Producer. Create a Pipeline that Connects to MySQL Consider…
Data Integration, meet Marketing Automation! Integrating MySQL with Marketo
StreamSets Data Collector’s HTTP pipeline stages allow a wide range of API integrations. I recently built a data pipeline to read customer data from a MySQL database: retrieving, creating and updating leads in Marketo via their REST API. I’ll share it with you today as a real-world application of data ingestion with StreamSets Data Collector. I’ve included videos that show…
