The DataOps Blog
Where Change Is Welcome
StreamSets Live: Demos with Dash!
45 minutes. Live demos. No (0) slides! Key takeaways: Visually build data pipelines like you code: build, verify, iterate, commit, rinse, repeat! Ingest data from…
Examples of Postgres CDC Pipelines for Amazon S3
This post is written by Marty Norris, Solutions Engineer, StreamSets. In this blog, you will learn how to ingest changed data from PostgreSQL WAL data and store it in Amazon Simple Storage Service (Amazon S3) using StreamSets Data Collector, a fast data ingestion engine. Consider the use case where you have data being constantly updated in your PostgreSQL database and…
How to Migrate to a Cloud Data Lake in Hours
Migrating from an on-premises data lake to a cloud data lake does not have to take months or even weeks. With intent-driven data pipelines, you can migrate your data from Hadoop to a cloud data lake in hours. In this blog post, I'll show you how to shift your focus from data movement to data value with a better way…
Apache Spark: Where Nothing Can Be Something
"Nothing is more real than nothing" - Samuel Beckett This post is written by Jeff Evans, Senior Software Engineer, StreamSets. While working on an issue for a prospect’s proof of concept pipeline using StreamSets Transformer, a modern Spark ETL engine, I came across a perplexing bug. A certain column in the data that -- by all accounts -- should have…
13 Data Engineering Best Practices At DNB
DNB is Norway's largest financial services group, and has a reputation as a trusted financial institution throughout the region. In this guest post, the DNB Data Engineering Centre of Practice team--Saleem Pothiwala, Operations Lead - Customer Insights, Jones Mabea Agwata, Software Engineer, and Bikram Rout, Data Engineer--share their best practices for harnessing the power of data for digital transformation. At…
Why Veterans Make Great Data Engineers
On this Veterans Day, we’d like to extend our gratitude to all of those who have served in the U.S. military. Thank you for your service and sacrifices. StreamSets is lucky to have Veterans on our team, and we’ve noticed some important overlap in skills for success in the military with those needed for success in data engineering. What is…
