The DataOps Blog
Where Change Is Welcome
StreamSets Live: Demos with Dash!
45 minutes. Live demos. No (0) slides! Key takeaways: Visually build data pipelines like you code: build, verify, iterate, commit, rinse, repeat! Ingest data from…
Sources And Destinations Podcast Episode #4 Joseph Arriola
Our analogy is, like when you have, one machine learning algorithm, the machine learning algorithm is training, training and training. So in the same way. We think together we are going to learn, learn, and learn with the different areas. It is really interesting because the data is the common factor. If you are in engineering, if you are in…
Load Change Data Capture Data from PostgreSQL to Amazon Redshift Using StreamSets
Change Data Capture is becoming essential to migrating to the cloud. In this blog, I have outlined detailed explanations and steps to load Change Data Capture (CDC) data from PostgreSQL to Amazon Redshift using StreamSets Data Collector, a fast data ingestion engine. The data pipeline first writes PostgreSQL CDC data to Amazon S3 and then executes a set of queries…
A Very Strange Year in the Data Business
A very, very long year ago, I wrote about three scenarios for adjusting your data practice with respect to the impact of the pandemic on their businesses, and data practices. With 12 months plus now behind us, and cautious optimism on the horizon, it’s time to look back at what predictions did and did not pan out, and new learnings from…
StreamSets Becomes a Certified HIPAA Business Associate
Data protection is an integral part of working with data today. 2020 saw a huge increase in records exposed in data breaches, and no company wants to leave themselves open to the reputation hit a breach causes. The stakes are even higher for industries that work with sensitive financial or patient data. In these cases, the government steps in with…
3 Important Trends in Data Integration
If you’d had a crystal ball this time last year, what would you have done differently? In the area of data integration, many of the changes that have taken place are accelerations of trends that were already underway. Even ahead of the crystal-clear proof of the changes needed, Gartner’s advice on data integration has been evolving to lead data and…