The DataOps Blog
Where Change Is Welcome
StreamSets Live: Demos with Dash!
45 minutes. Live demos. No (0) slides! Key takeaways: Visually build data pipelines like you code: build, verify, iterate, commit, rinse, repeat! Ingest data from…
How to Migrate to a Cloud Data Lake in Hours
Migrating from an on-premises data lake to a cloud data lake does not have to take months or even weeks. With intent-driven data pipelines, you can migrate your data from Hadoop to a cloud data lake in hours. In this blog post, I'll show you how to shift your focus from data movement to data value with a better way…
Apache Spark: Where Nothing Can Be Something
"Nothing is more real than nothing" - Samuel Beckett While working on an issue for a prospect’s proof of concept pipeline using StreamSets Transformer, a modern Spark ETL engine, I came across a perplexing bug. A certain column in the data that -- by all accounts -- should have been present was not. I finally isolated the problem in our…
13 Data Engineering Best Practices At DNB
DNB is Norway's largest financial services group, and has a reputation as a trusted financial institution throughout the region. In this guest post, the DNB Data Engineering Centre of Practice team--Saleem Pothiwala, Operations Lead - Customer Insights, Jones Mabea Agwata, Software Engineer, and Bikram Rout, Data Engineer--share their best practices for harnessing the power of data for digital transformation. At…
Why Veterans Make Great Data Engineers
On this Veterans Day, we’d like to extend our gratitude to all of those who have served in the U.S. military. Thank you for your service and sacrifices. StreamSets is lucky to have Veterans on our team, and we’ve noticed some important overlap in skills for success in the military with those needed for success in data engineering. What is…
Introducing Connection Catalog for Secure Data Access in StreamSets Control Hub
At StreamSets, we are thrilled to be releasing connection catalog in StreamSets Control Hub 3.19. Connections externalize the security credentials needed to connect to a system such as a database, a cloud platform such as AWS, Salesforce, or Kafka cluster, allowing platform operators to define a credential set once and securely share it with data pipeline developers. Developers can use…
