Nikolay Petrachkov (Nik for short) is a BI developer in Amsterdam by day, but in his spare time, he combines his passion for games and data engineering by building a project to analyze game-streaming data from Twitch. Nik discovered StreamSets Data Collector when he was looking for a way to build data pipelines to deliver insights […]
The term DataOps is a contraction of ‘Data Operations’ and a play on the idea of DevOps. It seems to have been coined in a 2015 blog post by Tamr co-founder and CEO Andy Palmer. In this blog post, I’ll dive into what DataOps means today, and how enterprises can adopt its practice to create […]
When it comes to creating and managing your dataflow pipelines, the graphical user interfaces of StreamSets Control Hub and StreamSets Data Collector put the complete power of our robust Data Operations Platform at your fingertips. There are times, however, when a more programmatic approach may be needed, and those times will be significantly more enjoyable […]
Today we are pleased to announce the general availability of StreamSets Control Hub version 3.2. StreamSets has built the industry’s only DataOps platform. We call it DataOps because our platform makes it easy to iteratively update dataflows when technology changes. Release 3.2 enhances manageability and repeatability between Control Hub instances which is critical when understanding […]
Hari Nayak’s recent blog post provides a quickstart for using StreamSets Control Hub to deploy multiple instances of StreamSets Data Collector on Google’s Kubernetes Engine (GKE). This post modifies the core scripts from that project in order to run on Minikube rather than GKE. As Minikube can run on a laptop, this allows lightweight testing of Control Hub’s Kubernetes support without needing to […]
Selvaraaju (‘Selva’) Murugesan is Senior Manager for Innovation and Data Analytics in the Australian Capital Territory (ACT) Government. Selva focuses on data management practices and data analytics, using StreamSets Data Collector to extract data from different databases, perform data cleansing on the fly and push data to the ACT Government’s Open Data Portal. Over the past few months, Selva has […]
Many StreamSets customers use Splunk to mine insights from machine-generated data such as server logs, but one problem they encounter with the default tools is that they have no way to filter the data that they are forwarding. While Splunk is a great tool for searching and analyzing machine-generated data, particularly in cybersecurity use cases, it’s easy […]
Spark Streaming + Data Collector + Secure Kafka When we first introduced cluster streaming mode with Apache Spark Streaming 1.3 and Apache Kafka 0.8 several years ago, Kafka didn’t support security features such as TLS (transport encryption, authentication) and Kerberos (authentication). In Spark 2.1, an updated Kafka connector was introduced with support for these features […]
Angel Alvarado is a senior software engineer at One Degree, a San Francisco-based non-profit, and also helps run the Molanco data engineering community. In his spare time, Angel enjoys playing Minecraft with his 11 year-old-cousin. Recently, Angel, found a fun way to combine his gaming with data engineering. This blog entry, reposted from the original with Angel’s […]
Detect, Secure and Govern Sensitive Data Upon Ingestion StreamSets is excited to announce a new product for protecting data in motion. StreamSets Data Protector, as the name would imply, extends the value of the StreamSets DataOps Platform to enable users to detect, secure and govern sensitive data as it flows around your business. Leveraging StreamSets’ […]
Join our mailing list to receive the latest news from StreamSets.
