The DataOps Blog

Where Change Is Welcome

Creating the OmniSci F1 Demo: Real-Time Data Ingestion With StreamSets

By May 8, 2019

Randy ZwitchRandy Zwitch is a Senior Director of Developer Advocacy at OmniSci, enabling customers and community users alike to utilize OmniSci to its fullest potential. With broad industry experience in energy, digital analytics, banking, telecommunications and media, Randy brings a wealth of knowledge across verticals as well as an in-depth knowledge of open-source tools for analytics. In this guest blog post, reposted from the original with permission, Randy explains the Formula 1 demo he built with StreamSets Data Collector to show real-time telemetry ingestion into OmniSci’s GPU-accelerated analytics platform.

DataOps in Healthcare

By August 28, 2018

In healthcare, data is delivering life-saving results with predictive capabilities that can address preventable outcomes. The intelligence guiding these initiatives relies on timely data delivery to applications and reviewers. This may involve complex, high velocity data forms with the expectation…

RingCentral Scales Out Big Data Streaming with StreamSets

By June 14, 2018

RingCentral is an award-winning global provider of cloud-unified communications and collaboration solutions. RingCentral solutions empower today’s mobile and distributed workforces to be connected anywhere and on any device through voice, video, team messaging, collaboration, SMS, conferencing, online meetings, contact center,…

Ingest Game-Streaming Data from the Twitch API

By May 25, 2018

Nick JastixNikolay Petrachkov (Nik for short) is a BI developer in Amsterdam by day, but in his spare time, he combines his passion for games and data engineering by building a project to analyze game-streaming data from Twitch. Nik discovered StreamSets Data Collector when he was looking for a way to build data pipelines to deliver insights from gaming data without having to write a ton of code. In this guest post, reposted from the original with his kind permission, Nik explains how he used StreamSets Data Collector to extract data about streams and games via the Twitch API. It’s a great example of applying enterprise dataops principles to a fun use case. Over to you, Nik…

A Fun Example of Streaming Data into Minecraft

By March 27, 2018

Angel AlvaradoAngel Alvarado is a senior software engineer at One Degree, a San Francisco-based non-profit, and also helps run the Molanco data engineering community. In his spare time, Angel enjoys playing Minecraft with his 11 year-old-cousin. Recently, Angel, found a fun way to combine his gaming with data engineering. This blog entry, reposted from the original with Angel’s kind permission, picks up the story…

Data Engineering can get really complex really quick and being aware of the hundreds of tools and data platforms in the industry can get very overwhelming. The following project is about how to use three data engineering tools to visualize data in a video game, it aims to solve a common data engineering problem with a twist to make it fun and entertaining.

Using StreamSets Control Hub for Scalable Deployment via Kubernetes

By January 15, 2018

StreamSets, Docker, KubernetesIn my previous blog entry, I explained how to spin up Data Collectors as Kubernetes deployments along with Dataflow Performance Manager. I recommended using a deployment with one replica as the design environment and a deployment with many replicas for execution. We recently announced StreamSets Control Hub which makes the Kubernetes integration way smoother! StreamSets Control Hub adds a Control Agent for Kubernetes that supports creating and managing Data Collector deployments and a Pipeline Designer that allows designing pipelines without having to install Data Collectors. In this blog, I will demonstrate how to take advantage of these features.

 

 

Streaming Data from Twitter for Analysis in Spark

By January 10, 2018

FootballHappy New Year! Our first blog entry of 2018 is a guest post from Josh Janzen, a data scientist based in Minnesota. Josh wanted to ingest tweets referencing NFL games into Spark, then run some analysis to look for a correlation between Twitter activity and game winners. Josh originally posted this entry on his personal blog, and kindly allowed us to repost it here. Over to you, Josh:

Tis the season of NFL football, and one way to capture excitement is Twitter data. I’ve tickered around with Twitter’s Developer API before, but this time I wanted to use a streaming product I’ve heard good things about: StreamSets Data Collector.

Back To Top