The DataOps Blog
Where Change Is Welcome
How to Become a Data Engineer (and Why You’re Lucky if You Do!)
If you’re interested in how to become a data engineer, you’re in luck. Data engineers are in high demand, and resources for how to become a data engineer abound! Data engineers are technical professionals who understand how data analysts and data scientists need data, then build the data pipelines to deliver the right data, in the right format, to the right…
Load Data from S3 to Snowflake and Use TensorFlow Model
Learn how to load data from S3 to Snowflake and serve a TensorFlow model in StreamSets Data Collector, a fast data ingestion engine, data pipeline for scoring on data flowing from S3 to Snowflake. Data and analytics are helping us become faster and smarter at staying healthy. Open data sets and analytics at cloud scale are key to unlocking the…
Data Engineers: Powerful Allies in Achieving an Equal Future
In Celebration of International Women’s Day It’s International Women’s Day, a day dedicated each year to helping girls and women achieve equality. Even from our privileged position in the tech capital of the world, women face inequality on executive teams and in boardrooms, albeit much more comfortably than most worldwide. But we also recognize that we have significant power to…
How To Load Data Into Google BigQuery on Dataproc and AutoML
Load Data Into Google BigQuery and AutoML In this blog, we will review ETL data pipeline in StreamSets Transformer, a Spark ETL engine, to ingest real-world data from Fire Department of New York (FDNY) stored in Google Cloud Storage (GCS), transform it, and load data in Google BigQuery curated. Once the transformed data is made available in Google BigQuery, it…
Optimizing Re-use for Data Quality to Scale Data Pipelines
As more data enters an organization’s ecosystem for transformation and is shared with more and more organizations both within and external to the business, data quality processes and frameworks are essential. Without data quality, users will lose trust in the analytics, resulting in stalled user adoption and analytic silos. Pulling all of the information together correctly creates symbiotic results that…
