The DataOps Blog
Where Change Is Welcome
StreamSets Live: Demos with Dash!
45 minutes. Live demos. No (0) slides! Key takeaways: Visually build data pipelines like you code: build, verify, iterate, commit, rinse, repeat! Ingest data from…
How To Load Data Into Google BigQuery on Dataproc and AutoML
What is Dataproc? Dataproc is a low-cost, Google Cloud Platform integrated, easy to use managed Spark and Hadoop service that can be leveraged for batch processing, streaming, and machine learning use cases. What is Google BigQuery? BigQuery is an enterprise grade data warehouse that enables high-performance SQL queries using the processing power of Google's infrastructure. Load Data Into Google BigQuery…
Optimizing Re-use for Data Quality to Scale Data Pipelines
As more data enters an organization’s ecosystem for transformation and is shared with more and more organizations both within and external to the business, data quality processes and frameworks are essential. Without data quality, users will lose trust in the analytics, resulting in stalled user adoption and analytic silos. Pulling all of the information together correctly creates symbiotic results that…
3 Reasons to Love Your Data Engineer
Data scientists get a lot of press these days, and it’s not without good reason. Companies live and die by data and decisions are made with the high visibility work that data scientists and analysts do. But these ‘front-end’ data professionals have a secret… Their job is infinitely more difficult without a data engineer in the background. According to CIO…
EDW or EDH? Data Lake, Warehouse or Lakehouse?
As John Zada puts it, “There can seldom be just one explanation to things. As a result our paradigms can be over-simplistic, incomplete or inaccurate – removing the complexity from the world which is actually one of its defining qualities.” So can we really evaluate such impactful tools in such simplistic terms? Database to Data Warehouse: How It All Began …
Does Migrating Workloads to AWS Require Re-writes?
Migrating your data workloads to AWS without redesign headaches -- urban developer lore or critical capability for modern enterprises? In this blog we will investigate by building a data pipeline to Google Cloud Platform and then migrating the workload to AWS. Moving clouds may sound like moving mountains. But your ability to move data quickly and reliably between clouds ensures…
