The DataOps Blog
Where Change Is Welcome
StreamSets Live: Demos with Dash!
45 minutes. Live demos. No (0) slides! Key takeaways: Visually build data pipelines like you code: build, verify, iterate, commit, rinse, repeat! Ingest data from…
Data Engineers: Powerful Allies in Achieving an Equal Future
In Celebration of International Women’s Day It’s International Women’s Day, a day dedicated each year to helping girls and women achieve equality. Even from our privileged position in the tech capital of the world, women face inequality on executive teams and in boardrooms, albeit much more comfortably than most worldwide. But we also recognize that we have significant power to…
How To Load Data Into Google BigQuery on Dataproc and AutoML
What is Dataproc? Dataproc is a low-cost, Google Cloud Platform integrated, easy to use managed Spark and Hadoop service that can be leveraged for batch processing, streaming, and machine learning use cases. What is Google BigQuery? BigQuery is an enterprise grade data warehouse that enables high-performance SQL queries using the processing power of Google's infrastructure. Load Data Into Google BigQuery…
Optimizing Re-use for Data Quality to Scale Data Pipelines
As more data enters an organization’s ecosystem for transformation and is shared with more and more organizations both within and external to the business, data quality processes and frameworks are essential. Without data quality, users will lose trust in the analytics, resulting in stalled user adoption and analytic silos. Pulling all of the information together correctly creates symbiotic results that…
3 Reasons to Love Your Data Engineer
Data scientists get a lot of press these days, and it’s not without good reason. Companies live and die by data and decisions are made with the high visibility work that data scientists and analysts do. But these ‘front-end’ data professionals have a secret… Their job is infinitely more difficult without a data engineer in the background. According to CIO…
EDW or EDH? Data Lake, Warehouse or Lakehouse?
As John Zada puts it, “There can seldom be just one explanation to things. As a result our paradigms can be over-simplistic, incomplete or inaccurate – removing the complexity from the world which is actually one of its defining qualities.” So can we really evaluate such impactful tools in such simplistic terms? Database to Data Warehouse: How It All Began …
