Where Change Is Welcome
Data pipelines are meant to transfer data from a source or legacy system to a target system. Easy right? Well not so much. As a Data Engineer, it’s our job to be responsible for multiple different data pipeline architecture decisions…
Your company may not have a documented or formally defined data governance framework. But if data is created and used in your organization, you have a governance framework. Whether it’s effective or not… that’s another question. The challenge with data…
Successful organizations learn from the past, are able to predict the future, and understand the shape of their present. Enterprise Data Integration is the magic that binds all these views together. What is Enterprise Data Integration? Enterprise Data Integration is…
Looking to build a Redis pipeline? Before we dig in and build our pipeline, let’s discuss what Redis is, its benefits, and common Redis pipeline use cases. What is Redis? Redis, which stands for Remote Dictionary Server, is an open…
Author Bio: Isaac Omolayo is a Data Engineer with 4 years of experience in building ETL processes, data infrastructures, automated data systems, working with cloud engines, and data analytics. He has vast knowledge in working with Apache Kafka, Kafka Streaming,…
The relationship of data to today’s modern enterprise is becoming more sophisticated. As data sources continue to increase and data formats continue to evolve, ETL and its place in modern data integration are also changing. What was once a simple…
MLOps is emerging now more than ever in everyday workflows. Data Engineers are often tasked with the plumbing work to make data available for machine learning models. Even though statistically a small number of models get deployed to production, the…
The list of issues that can derail a data migration can be scary. This is why, as far back as 2009, and as recently as 2021, Gartner has released statistics detailing the prevalence of failed, delayed, and over budget data…
Organizations today use a ton of cloud services. Estimated averages range anywhere from a couple hundred per organization to almost 1300 per. Most of these come in the form of SaaS apps, all of which contain valuable data. On top…
Over the past 20 years the relationship between hardware, software and data has changed dramatically. The consequences for your business have too. The simplicity of hardware, software, and a tiny sliver of data in the middle, contained within a single…