The DataOps Blog
Where Change Is Welcome
StreamSets Live: Demos with Dash!
45 minutes. Live demos. No (0) slides! Key takeaways: Visually build data pipelines like you code: build, verify, iterate, commit, rinse, repeat! Ingest data from…
Alexa, Start My Data Pipeline
Imagine asking Amazon Alexa or Google Home to run your ETL, data processing, and machine learning data pipelines. For example, "Start my data pipeline on Amazon EMR", “How many active jobs do I have running on Databricks?", or "Stop my machine learning pipeline on Snowflake" — all without firing up your computer or asking an admin or your favorite data…
Data Engineering with a Purpose: Tackling Climate Change
Data has become one of the most powerful tools at humanity’s disposal, and it can be used for a variety of end purposes. As data experts, data engineers can take pride in their mastery of the many technical skills and practices necessary to deliver data to those who need it. Many data engineers are further inspired by the broader impact…
Sources And Destinations Podcast Episode #4 Joseph Arriola
Our analogy is, like when you have, one machine learning algorithm, the machine learning algorithm is training, training and training. So in the same way. We think together we are going to learn, learn, and learn with the different areas. It is really interesting because the data is the common factor. If you are in engineering, if you are in…
Load Change Data Capture Data from PostgreSQL to Amazon Redshift Using StreamSets
Change Data Capture is becoming essential to migrating to the cloud. In this blog, I have outlined detailed explanations and steps to load Change Data Capture (CDC) data from PostgreSQL to Amazon Redshift using StreamSets Data Collector, a fast data ingestion engine. The data pipeline first writes PostgreSQL CDC data to Amazon S3 and then executes a set of queries…
A Very Strange Year in the Data Business
A very, very long year ago, I wrote about three scenarios for adjusting your data practice with respect to the impact of the pandemic on their businesses, and data practices. With 12 months plus now behind us, and cautious optimism on the horizon, it’s time to look back at what predictions did and did not pan out, and new learnings from…