skip to Main Content

The DataOps Blog

Where Change Is Welcome

Data Pipeline Architecture: Key Design Principles & Considerations for Data Engineers

Engineering

Data pipelines are meant to transfer data from a source or legacy system to a target system. Easy right? Well not so much. As a Data Engineer, it’s our job to be responsible for multiple different data pipeline architecture decisions during the design phase. Answering questions like, what are the source/s and target/s for this data? Is this data coming…

Drew-Nicolette-Data-Engineer By May 20, 2022

Thinking Through the Basics of Your Data Governance Framework

Engineering

Your company may not have a documented or formally defined data governance framework. But if data is created and used in your organization, you have a governance framework.  Whether it’s effective or not… that’s another question.  The challenge with data governance frameworks, and data governance in general, is tying together the elements of how your organization collects, manages, and archives…

By May 19, 2022

Enterprise Data Integration: What it is, Why it Matters, and How to Approach It

Engineering

Successful organizations learn from the past, are able to predict the future, and understand the shape of their present. Enterprise Data Integration is the magic that binds all these views together. What is Enterprise Data Integration? Enterprise Data Integration is the process of combining data from various sources and unifying that data in a sensible way. In the past, this…

Brenna Buuck By May 18, 2022

Redis Pipeline: How to Publish and Subscribe Data from Redis to Your Destination

Engineering

Looking to build a Redis pipeline? Before we dig in and build our pipeline, let’s discuss what Redis is, its benefits, and common Redis pipeline use cases. What is Redis? Redis, which stands for Remote Dictionary Server, is an open source (BSD licensed), in-memory data structure store used as a database, cache, message broker, and streaming engine. Redis delivers sub-millisecond…

Kafka Streaming: Live Streaming Kafka Application to Cassandra

Engineering, Use Cases

Author Bio: Isaac Omolayo is a Data Engineer with 4 years of experience in building ETL processes, data infrastructures, automated data systems, working with cloud engines, and data analytics. He has vast knowledge in working with Apache Kafka, Kafka Streaming, Apache Spark, Apache Airflow, Databases (SQL/NoSQL), Python and many other Big Data technologies. Generally, Isaac is a lover of open-source…

By May 12, 2022
Back To Top

We use cookies to improve your experience with our website. Click Allow All to consent and continue to our site. Privacy Policy