The DataOps Blog
Where Change Is Welcome
Chat with the StreamSets Team via Slack!
Since its inception last October, the sdc-user Google group has been the primary medium for you, our community, to communicate with us, the StreamSets Team. We've seen over a thousand messages, and participated in discussions around installation, configuration, bugs, feature requests, and every other aspect of StreamSets Data Collector. While sdc-user works well for many interactions, it does lack the…
Retrieving Metrics via the StreamSets Data Collector REST API
Last week, I explained how I was able to run StreamSets Data Collector Engine on a Raspberry Pi 3, ingesting sensor data and writing it to Cassandra. With that working, I wanted to show pipeline metrics across data pipelines on Adafruit's awesome PiTFT Plus 2.8" screen. In this blog post, I'll explain how I was able to write a Python…
Announcing Data Collector ver 1.5.0.0
We are excited to announce the release of the next version of StreamSets Data Collector. With this release we have a number of new features and enhancements and 40+ bug fixes. Automatic creation and updates to Hive schemas. This new functionality automatically creates schemas for Hive tables, and if it detects schema changes in the incoming data set (schema drift)…
Struggling with Bad Data? What to Do From the Enterprise
Last week we announced the results of a survey of over 300 enterprise data professionals conducted by Dimensional Research and sponsored by StreamSets. We were trying to understand the market’s state of play for managing their big data flows. What we discovered was that there is an alarming issue at hand: companies are struggling to detect and keep bad data…
Ingesting Sensor Data on the Raspberry Pi with StreamSets Data Collector
In the unlikely event you're not familiar with the Raspberry Pi, it's an ARM-based computer about the same size as a deck of playing cards. The latest iteration, Raspberry Pi 3, has a 1.2GHz ARMv8 CPU, 1MB of memory, integrated Wi-Fi and Bluetooth, all for the same $35 price tag as the original Raspberry Pi released in 2012. Running a…