Use Cases Archives - StreamSets

Read and Write JSON to MapR DB with StreamSets Data Collector

March 5, 2017 Rupal Shah Engineering, StreamSets News, Use Cases

MapR DB logo MapR-DB is an enterprise-grade, high performance, NoSQL database management system. As a multi-model NoSQL database, it supports both JSON document models and wide column data models. MapR-DB stores JSON documents in tables; documents within a table in MapR-DB can have different structures. StreamSets Data Collector enables working with MapR-DB documents with its powerful schema-on-read and ingestion capability.

With StreamSets Data Collector, I’ll show you how easy it is to stream data from MongoDB into a MapR-DB table as well as stream data out of the MapR-DB table into MapR Streams.

The Challenge of Fetching Data for Apache Spot (incubating)

October 19, 2016 Rick Bilodeau Industry, Use Cases

Reposted from the Cloudera Vision blog.

What do Sony, Target and the Democratic Party have in common?

Besides being well-respected brands, they’ve all been subject to some very public and embarrassing hacks over the past 24 months. Because cybercrime is no longer driven by angst-ridden teenagers but rather professional criminal organizations and state-sponsored hacker groups, the halcyon days of looking for a threat signatures are well behind us.

Creating a Post-Lambda World with Apache Kudu

September 23, 2016 Arvind Prabhakar Engineering, Industry, Use Cases

Apache Kudu and Open Source StreamSets Data Collector Simplify Batch and Real-Time Processing

As originally posted on the Cloudera VISION Blog.

At StreamSets, we come across dataflow challenges for a variety of applications. Our product, StreamSets Data Collector is an open-source any-to-any dataflow system that ensures that all your data is safely delivered in the various systems of your choice. At its core is the ability to handle data drift that allows these dataflow pipelines to evolve with your changing data landscape without incurring redesign costs.

This position at the front of the data pipeline has given us visibility into various use cases, and we have found that many applications rely on patched-together architectures to achieve their objective.

The Complementary Nature of Data Ingestion and Data Preparation

May 25, 2016 Rick Bilodeau Industry, Use Cases

I am always eager to learn about new architectures and best big data practices. Recently I came across a paper from Trifacta discussing the role of data preparation and it got me thinking about the complementary nature of data ingestion and data preparation.

Data preparation, more colorfully known as data wrangling, is the activity performed by data-driven professionals, such as data or business analysts, to explore, clean, transform and blend data of all varieties to make it trustworthy for analysis or predictive modeling. A form of data manipulation that has traditionally been achieved using Excel or, for more technically-advanced end users, languages such as R, SAS or Python. But with the rise of enormous and dynamic data sets in Hadoop, these approaches are no longer feasible. Trifacta took the lead in creating a self-service web-based solution that enables business users to access and manipulate data stored in Hadoop without needing programming skills.

How Trend Micro Uses StreamSets – An Interview with the Threat Research Team

March 21, 2016 Kirit Basu Industry, StreamSets News, Use Cases

The Forward-Looking Threat Research team at Trend Micro were early adopters of StreamSets Data Collector. They use StreamSets to ingest data from a wide variety of sources to create a Threat Assessment Dashboard in Elasticsearch. In this interview, we talk with members of their team about how they evaluated StreamSets and implemented it in their production environment in a short period of time.

Binlog Processing Using Maxwell, Kafka & StreamSets

March 2, 2016 Rick Bilodeau Engineering, Use Cases

This is a nice example of Kafka enablement using Maxwell (a mysql-to-kafka binlog processor) and StreamSets Data Collector from the folks at B23. It includes a schema change listener for handling data drift. Enjoy!

Innovate on Your Data – Maxwell Meets StreamSets

Simple Kafka Enablement Using StreamSets (video)

February 24, 2016 Kirit Basu Industry, Use Cases, Videos

You can simplify use of Kafka within your infrastructure using StreamSets Data Collector. Watch this short step-by-step tutorial to learn how.

https://vimeo.com/153061876

How-to: Build a Real-Time Search System using StreamSets, Apache Kafka, and Cloudera Search

February 17, 2016 Jonathan Natkins Engineering, StreamSets News, Use Cases

The following was re-posted from the Cloudera Engineering Blog.

Thanks to Jonathan Natkins, a field engineer from StreamSets, for the guest post below about using StreamSets Data Collector—open source, GUI-driven ingest technology for developing and operating data pipelines with a minimum of code—and Cloudera Search and HUE to build a real-time search environment.

Continuous Ingest to Elasticsearch (video)

February 13, 2016 Kirit Basu Industry, Use Cases, Videos

You can send log files to Elasticsearch using StreamSets Data Collector. Watch this short step-by-step to see how.

https://vimeo.com/152097120

StreamSets Monitoring with Grafana, InfluxDB, and jmxtrans

January 14, 2016 Adam Kunicki Engineering, Use Cases

The ability to monitor your critical infrastructure is a must, and we designed the StreamSets Data Collector (SDC) with this in mind: metrics are exposed through both the REST API and JMX. While there are many approaches to monitoring these metrics, let’s walk through a specific end-to-end example using jmxtrans to collect metrics, InfluxDB to store them, and Grafana to visualize them.

Use Cases