The DataOps Blog
Where Change Is Welcome
StreamSets Live: Demos with Dash!
45 minutes. Live demos. No (0) slides! Key takeaways: Visually build data pipelines like you code: build, verify, iterate, commit, rinse, repeat! Ingest data from…
Using IAM Roles with StreamSets Data Collector on AWS Elastic Kubernetes Service
This post describes how StreamSets Data Collector, a fast and flexible data ingestion engine, can be configured to take advantage of IAM Roles when running on AWS Elastic Kubernetes Service (EKS). IAM Roles allow data engineers to concentrate on business logic while AWS permissions are managed independently by AWS Administrators in accordance with AWS Best Practices for Security. If Data…
Model Experiments, Tracking and Registration using MLflow on Databricks
Learn how StreamSets, a modern data integration platform for DataOps, can help expedite operations at some of the most crucial stages of Machine Learning Lifecycle and MLOps. Data Acquisition And Preparation Machine learning models are only as good as the quality of data and the size of datasets used to train the models. Data has shown that data scientists spend…
What are Grok Patterns?
Grok leverages regular expression language that allows you to name existing patterns and/or combine them into more complex patterns. Because Grok is based on regular expressions, any valid regular expressions (regexp) are also valid in grok. In StreamSets Data Collector, a fast data ingestion engine, you can use a single grok pattern, compose several patterns, or create a custom pattern to…
Best Practices For Running StreamSets Data Collector On Kubernetes
An increasing number of StreamSets' customers are deploying Data Collector on Kubernetes, taking advantage of the reliability and elastic scaling Kubernetes provides. This post describes best practices for configuring and managing Data Collector on Kubernetes. More information and in-depth examples of the techniques below can be found in StreamSets' Kubernetes Deployment Tutorial. 1) Use Control Hub to manage Data Collector…
Spark ETL To Derive Sales Insights on Azure HDInsight And Power BI
In this blog, we will review how easy it is to set up an end-to-end ETL data pipeline that runs on StreamSets Transformer to perform extract, transform, and load (ETL) operations. The pipeline uses Apache Spark for Azure HDInsight cluster to extract raw data and transform it (cleanse and curate) before storing it in multiple destinations for efficient downstream analysis.…