Where Change Is Welcome
Every data-driven enterprise looking to get the most out of its data has a continuous hunt for a cost-effective, scalable, high-performance storage and analytics architectural solution. Organizations have moved from traditional data warehouses to data lakes and are now shifting…
A successful data migration strategy involves moving data from one source to another with as little friction as possible. That friction usually comes in cost, data loss, or downtime accessing the target or destination data sources. A good migration design…
With data-driven organizations 23 times more likely to acquire new customers and six times more likely to retain them, it’s no wonder most companies are shifting to a data-driven approach. This shift creates an ever-growing need for clean, current, and…
In the business world today, a single customer may interact with businesses at various touchpoints. Their data may span multiple locations like hybrid cloud environments, databases, CRM systems, and web and mobile applications. Managing and accessing these data for business…
A while ago we talked to a prospect that wanted to stream their IoT data into Delta Lake without a Spark (Databricks) cluster running continuously. While streaming data into Delta Lake is supported by StreamSets, the second part of this…
Data warehouses have become essential for organizations dealing with massive amounts of data, and while other alternatives come to market their adoption remains on the rise. This increased adoption accounts for a projected Compound Annual Growth Rate (CAGR) of 10.7%…
While some might observe that the difference between slowly changing dimensions (SCD) And Change Data Capture (CDC) might be subtle, there is in fact a technical difference between the two processes. Both processes detect changes in a source database and…
An organization rarely has a single data source. Instead, it aggregates data from various sources like websites, applications, social media, and databases to make data easily accessible through data integration. In addition, this data needs to be transformed before being…
Data is everywhere, in different formats and databases. Being able to integrate multiple, highly varied data sources is essential to running a business today. You have to be able to Extract-Transfer-Load (ETL) the data from each source into a database…
Whether you are building your very first pipeline or you’re an old pro, these best practices for building data pipelines can help you make pipelines that are easy to understand and therefore easy to maintain and extend. Design Data Pipelines…