The DataOps Blog
Where Change Is Welcome
Visualize StreamSets Data Collector Metrics with Datadog
Back in January, Adam blogged about StreamSets Monitoring with Grafana, InfluxDB, and jmxtrans. While the Grafana/InfluxDB/jmxtrans open source stack works great, there's quite a lot of setup and configuration to keep track of. Another tool that StreamSets customers are using to monitor their infrastructure is Datadog; in today's blog post I'll show you the basics of configuring Datadog with StreamSets…
Ingesting MQTT Traffic into Riak TS via RabbitMQ and StreamSets
Riak KV is an open source, distributed, NoSQL key-value data store oriented towards high availability, fault tolerance and scalability. With its initial release in 2009, Risk KV is in use at companies such as AT&T, Comcast and GitHub. Last October, Basho, the vendor behind Riak KV, announced Riak TS. Riak TS, another distributed, NoSQL data store, is optimized for time…
Analyzing Salesforce Data with StreamSets, Elasticsearch, and Kibana
UPDATE - Salesforce origin and destination stages, as well as a destination for Salesforce Wave Analytics, were released in StreamSets Data Collector 2.2.0.0. Use the supported, shipping Salesforce stages rather than the unsupported code mentioned below! After I published a proof-of-concept Salesforce Origin for StreamSets Data Collector (SDC), I noticed an article on the Elastic blog, Analyzing Salesforce Data with Logstash,…
Announcing Data Collector ver 1.4.0.0
We are excited to announce the release of the next version of StreamSets Data Collector. With this release we have a number of new features and enhancements and 60+ bug fixes. New Features : SFTP/FTP origin to read files from an SFTP or FTP server. MapR DB destination to write data to MapR DB using the HBase API. XML Flattener…
The Complementary Nature of Data Ingestion and Data Preparation
I am always eager to learn about new architectures and best big data practices. Recently I came across a paper from Trifacta discussing the role of data preparation and it got me thinking about the complementary nature of data ingestion and data preparation. Data preparation, more colorfully known as data wrangling, is the activity performed by data-driven professionals, such as…
