The DataOps Blog
Where Change Is Welcome
StreamSets Live: Demos with Dash!
45 minutes. Live demos. No (0) slides! Key takeaways: Visually build data pipelines like you code: build, verify, iterate, commit, rinse, repeat! Ingest data from…
3 AWS Data Pipelines to Impress Your Boss
In this blog, you will learn how to design AWS Data Pipelines in StreamSets Data Collector, a fast data ingestion engine, that you can then use to impress your boss with in 2021 -- the year of the data engineer! Migrate An On-Prem Oracle Database to AWS Aurora RDS Consider the use case where you have a legacy data warehouse…
2021: The Year of the Data Engineer
2021 is going to be a good year. We may not be there quite yet, but there’s no denying things are looking up. As vaccines kick in across the world, focus will shift from fighting for survival to reopening, rebuilding, and strengthening fractured economies alongside our returning health. Remember our 3 scenarios for adjusting your data practice? It's clear that…
StreamSets for Data Engineers: Year 2020 In Review
Times are definitely not “normal”, and more challenging for some than others, but I hope everyone is continuing to stay safe and I wish everyone a very happy holiday season. More importantly, I also want to take this opportunity and thank our community members and customers because despite what might have been going on in everyone's lives during this pandemic,…
Send Kafka Messages To Amazon S3
In this post, we will take a look at best practices for integrating StreamSets Data Collector (SDC), a fast data ingestion engine, with Kafka, and take a deep dive into the details with regards to sending Kafka messages to Amazon S3. Apache Kafka is a distributed event streaming platform used by thousands of companies for streaming analytics and for other…
Examples of Postgres CDC Pipelines for Amazon S3
In this blog, you will learn how to ingest changed data from PostgreSQL WAL data and store it in Amazon Simple Storage Service (Amazon S3) using StreamSets Data Collector, a fast data ingestion engine. Consider the use case where you have data being constantly updated in your PostgreSQL database and you want to extract the changed data in real-time and…
