Where change is welcome.
StreamSets Data Collector (SDC) supports 69 sources, including relational and no-SQL databases, on-prem and cloud file systems and a handful of messaging applications (documentation). Yet, occasionally, customers ask if Data Collector can integrate with an app or system not explicitly…
Quality data is key in decision-making. But, demanding big data to be processed in seconds creates a lot of pressure on data engineering systems and can impact the accuracy of the processed data. Large datasets are usually processed by batch…
Data is everywhere, in different formats and databases. Being able to integrate multiple, highly varied data sources is essential to running a business today. You have to be able to Extract-Transfer-Load (ETL) the data from each source into a database…
Using StreamSets DataOps Platform To Integrate Data from PostgreSQL to AWS S3 and Redshift: A Reference Architecture This document describes the reference architecture for integrating data from a database to Amazon Web Services (AWS) data analytics stack utilizing the StreamSets…
There are a couple of different approaches to SQL Server replication. I’ve always taken the common approach, one you’re likely very familiar with, Change Data Capture (CDC). In fact, I’ve written guides on CDC replication of other databases using StreamSets…
No one can argue the value streaming data provides for an organization. From analyzing data while in motion, it’s no surprise that data-driven organizations are incorporating it into their overall data processes to help identify business strategies that work and…
You may be taking an interest in Snowpark UDFs as a way to take full advantage of the power of your Snowflake Data Cloud. You’re not alone. Companies are always looking for new ways to get more out of their…
We recently kicked off our Women in DataOps series with Dr. Beverly Wright, a data and analytics thought leader with 30 years of experience. Beverly has spent many years teaching data science and analytics to undergraduates, master’s students, PhDs, and…
Author Bio: Isaac Omolayo is a Data Engineer with 4 years of experience in building ETL processes, data infrastructures, automated data systems, working with cloud engines, and data analytics. He has vast knowledge in working with Apache Kafka, Kafka Streaming,…
Maryann Agofure is a Cloud engineer and technical writer who loves fried plantains. She’s currently on a cloud data engineering journey to gain a deeper understanding of data pipelines for better analysis, hence her interest in data orchestration, and seeking…