The DataOps Blog
Where Change Is Welcome
Data Pipeline Architecture: Key Design Principles & Considerations for Data Engineers
Data pipelines are meant to transfer data from a source or legacy system to a target system. Easy right? Well not so much. As a Data Engineer, it’s our job to be responsible for multiple different data pipeline architecture decisions during the design phase. Answering questions like, what are the source/s and target/s for this data? Is this data coming…
Thinking Through the Basics of Your Data Governance Framework
Your company may not have a documented or formally defined data governance framework. But if data is created and used in your organization, you have a governance framework. Whether it’s effective or not… that’s another question. The challenge with data governance frameworks, and data governance in general, is tying together the elements of how your organization collects, manages, and archives…
Enterprise Data Integration: What it is, Why it Matters, and How to Approach It
Successful organizations learn from the past, are able to predict the future, and understand the shape of their present. Enterprise Data Integration is the magic that binds all these views together. What is Enterprise Data Integration? Enterprise Data Integration is the process of combining data from various sources and unifying that data in a sensible way. In the past, this…
Redis Pipeline: How to Publish and Subscribe Data from Redis to Your Destination
Looking to build a Redis pipeline? Before we dig in and build our pipeline, let’s discuss what Redis is, its benefits, and common Redis pipeline use cases. What is Redis? Redis, which stands for Remote Dictionary Server, is an open source (BSD licensed), in-memory data structure store used as a database, cache, message broker, and streaming engine. Redis delivers sub-millisecond…
Kafka Streaming: Live Streaming Kafka Application to Cassandra
Author Bio: Isaac Omolayo is a Data Engineer with 4 years of experience in building ETL processes, data infrastructures, automated data systems, working with cloud engines, and data analytics. He has vast knowledge in working with Apache Kafka, Kafka Streaming, Apache Spark, Apache Airflow, Databases (SQL/NoSQL), Python and many other Big Data technologies. Generally, Isaac is a lover of open-source…
