skip to Main Content

The DataOps Blog

Where Change Is Welcome

How To Load Data Into Google BigQuery on Dataproc and AutoML

Engineering

What is Dataproc? Dataproc is a low-cost, Google Cloud Platform integrated, easy to use managed Spark and Hadoop service that can be leveraged for batch processing, streaming, and machine learning use cases. What is Google BigQuery? BigQuery is an enterprise grade data warehouse that enables high-performance SQL queries using the processing power of Google's infrastructure. Load Data Into Google BigQuery…

By February 23, 2021

Optimizing Re-use for Data Quality to Scale Data Pipelines

StreamSets News

As more data enters an organization’s ecosystem for transformation and is shared with more and more organizations both within and external to the business, data quality processes and frameworks are essential. Without data quality, users will lose trust in the analytics, resulting in stalled user adoption and analytic silos. Pulling all of the information together correctly creates symbiotic results that…

3 Reasons to Love Your Data Engineer

StreamSets News

Data scientists get a lot of press these days, and it’s not without good reason. Companies live and die by data and decisions are made with the high visibility work that data scientists and analysts do. But these ‘front-end’ data professionals have a secret… Their job is infinitely more difficult without a data engineer in the background.  According to CIO…

By February 12, 2021

EDW or EDH? Data Lake, Warehouse or Lakehouse?

StreamSets News

As John Zada puts it, “There can seldom be just one explanation to things. As a result our paradigms can be over-simplistic, incomplete or inaccurate – removing the complexity from the world which is actually one of its defining qualities.” So can we really evaluate such impactful tools in such simplistic terms? Database to Data Warehouse: How It All Began …

By February 4, 2021

Does Migrating Workloads to AWS Require Re-writes?

Engineering

Migrating your data workloads to AWS without redesign headaches -- urban developer lore or critical capability for modern enterprises? In this blog we will investigate by building a data pipeline to Google Cloud Platform and then migrating the workload to AWS.  Moving clouds may sound like moving mountains. But your ability to move data quickly and reliably between clouds ensures…

Back To Top

We use cookies to improve your experience with our website. Click Allow All to consent and continue to our site. Privacy Policy