Where change is welcome.
While some might observe that the difference between slowly changing dimensions (SCD) And Change Data Capture (CDC) might be subtle, there is in fact a technical difference between the two processes. Both processes detect changes in a source database and…
An organization rarely has a single data source. Instead, it aggregates data from various sources like websites, applications, social media, and databases to make data easily accessible through data integration technologies. In addition, this data needs to be transformed before…
Whether you are building your very first pipeline or you’re an old pro, these best practices for building data pipelines can help you make pipelines that are easy to understand and therefore easy to maintain and extend. Design Data Pipelines…
Data lacks value until organizations can gain business intelligence and insights from it. The ability to transform and maximize the value of an organization's data can be challenging for most businesses. Data storage options must hold company data and be…
Using StreamSets Data Integration Platform To Integrate Data from PostgreSQL to AWS S3 and Redshift: A Reference Architecture This document describes the reference architecture for integrating data from a database to Amazon Web Services (AWS) data analytics stack utilizing the…
Instant notifications, product recommendations and updates, and fraud detection are practical use-cases of stream processing. With stream processing, data streaming and analytics occur in real-time, which helps drive fast decision-making. However, building an effective streaming architecture to handle data needs…
This blog post was updated March 15, 2023 Data warehouse design is the process of defining how the interdependent systems and processes involved in data warehousing will be implemented to align business needs and functional requirements. The data warehouse design…
This blog post was updated March 14, 2023 Data warehouse schema is a description, represented by objects such as tables and indexes, of how data relates logically within a data warehouse. Star, galaxy, and snowflake schema are types of warehouse…
The modern data stack is less like a stack and more like an ecosystem with many participants. This constellation of technologies coalesces around a few guiding principles. Three Guiding Principles The first principle of the modern data stack is complete…
Successful organizations learn from the past, are able to predict the future, and understand the shape of their present. Enterprise Data Integration is the magic that binds all these views together. What is Enterprise Data Integration? Enterprise Data Integration is…