Where change is welcome.
StreamSets has added support for Oracle Data Guard to one of our most popular origins: Oracle Change Data Capture (CDC) What Is Oracle Data Guard? Oracle Data Guard is Oracles’ database disaster recovery solution. Oracle Data Guard allows for the…
This post was updated on February 24, 2023 The best way to understand something is through concrete examples. I’ve put together seven examples of data pipelines that represent very typical patterns that we see our customers engage in. These are…
While some might observe that the difference between slowly changing dimensions (SCD) And Change Data Capture (CDC) might be subtle, there is in fact a technical difference between the two processes. Both processes detect changes in a source database and…
The digital world moves fast. To keep up with customers (and competitors), organizations need to make split-second decisions and take action in real-time. Because as Nucleus Research pointed out, data has a half-life when it comes to tactical, operational, and…
Imagine asking Amazon Alexa or Google Home to run your ETL, data processing, and automate your data pipelines. For example, "Start my data pipeline on Amazon EMR", “How many active jobs do I have running on Databricks?", or "Stop my…
Change Data Capture is becoming essential to migrating to the cloud. In this blog, I have outlined detailed explanations and steps to load Change Data Capture (CDC) data from PostgreSQL to Redshift using StreamSets Data Collector, a fast data ingestion…
In this blog, you will learn how to ingest changed data from PostgreSQL WAL data and store it in Amazon Simple Storage Service (Amazon S3) using a Postgres CDC pipeline in StreamSets. Consider the use case where you have data…
In this post, we will explore how to bulk ingest and process change data capture (CDC) information from Oracle 19c database using the enhanced Oracle CDC Client origin into Databricks Delta Lake in StreamSets Data Collector, a fast data ingestion…
Yannick Jaquier is a Database Technical Architect at STMicroelectronics in Geneva, Switzerland. Recently, Yannick started experimenting with StreamSets Data Collector Engine‘s Oracle CDC Client origin, building pipelines for Oracle replication–replicating data to a second Oracle database, a JSON file, and a MySQL database. Yannick documented his journey very thoroughly, and kindly gave us permission to repost his findings from his original blog entry.
Today’s post is from Raphaël Velfre, a senior data engineer at MapR. Raphaël has spent some time working with StreamSets Data Collector (SDC) and MapR’s Converged Data Platform. In this blog entry, Raphaël explains how to use SDC for MySQL CDC to extract data from MySQL and write it to MapR Streams, and then move data from MapR Streams to MapR-FS via SDC, where it can be queried with Apache Drill.