The DataOps Blog
Where Change Is Welcome
Announcing StreamSets Data Collector 3.16.0
StreamSets is excited to announce the immediate availability of StreamSets Data Collector 3.16.0. StreamSets Data Collector is a powerful design and execution engine. It enables you to build and deploy intent driven and data drift resilient smart data pipelines. Highlights There are some great new features and enhancements included in this release—below I’ve reviewed some of the highlights. For a…
Using Kubernetes Secrets in Data Collector Pipelines
In addition to StreamSets Data Collector's support for a variety of Credential Stores, one can also use Kubernetes Secrets as a mechanism for securely managing credentials and environment-specific properties within Data Collector pipelines. This post shows an example of using a Kubernetes Secret to manage MySQL connection properties for a JDBC Producer. Create a Pipeline that Connects to MySQL Consider…
Data Integration, meet Marketing Automation! Integrating MySQL with Marketo
StreamSets Data Collector’s HTTP pipeline stages allow a wide range of API integrations. I recently built a data pipeline to read customer data from a MySQL database: retrieving, creating and updating leads in Marketo via their REST API. I’ll share it with you today as a real-world application of data integration with StreamSets Data Collector. I’ve included videos that show…
Ingesting AWS Marketplace Analytics Data into MySQL
StreamSets Data Collector is available in the major cloud marketplaces; as an AWS Marketplace vendor, StreamSets has access to a number of data sets detailing product usage and billing. Data sets are available interactively via a web portal and via the AWS Marketplace Commerce Analytics Service (MCAS) API. While the individual reports are useful, it would be even more valuable…
Clickstream Analysis on Amazon EMR, Amazon Redshift and Elasticsearch
Learn how quickly you can start ingesting and aggregating clickstream logs using StreamSets Transformer running on Amazon EMR, and see how the data is analyzed in Elasticsearch, Kibana, and Amazon Redshift. Clickstream analysis by definition is the process of collecting, analyzing, and reporting aggregate information about webpage visits. StreamSets Transformer is an execution engine that runs on different "flavors" of…