Apache Pulsar is an open-source distributed pub-sub messaging system originally created at Yahoo, and a top-level Apache project since September 2018. StreamSets Data Collector 3.5.0, released soon after, introduced the Pulsar Consumer and Producer pipeline stages. In this blog entry I’ll explain how to get started creating dataflow pipelines for Pulsar.
In my previous blog, we looked at using TensorFlow models in dataflow pipelines to generate predictions and classifications in real-time. In this blog post, I will walk you through using Databricks ML models in StreamSets Data Collector for low-latency inference.
Last week, a sizable group of professionals ascended on Pier 27 in San Francisco for the 3rd Annual Kafka Summit. StreamSets was excited to support the event, especially because of the focus our customers have on Kafka as a critical mechanism for delivering on real-time capabilities. Key themes for this year’s summit included Microservices, Kubernetes, […]
The real value of a modern DataOps platform is realized only when business users and applications are able to access raw and aggregated data from a range of sources, and produce data-driven insights in a timely manner. And with Machine Learning (ML), analysts and data scientists can leverage historical data to help make better, data-driven […]
StreamSets is pleased to announce a new partnership with Oracle Cloud Infrastructure (OCI). As enterprises move their big data workloads to the cloud, it becomes imperative that their Data Operations are more resilient and adaptive to continue to serve the business’s needs. This is why StreamSets Data Collector™ is now easily deployable on OCI. What […]
Apache Flume and Apache Sqoop were the tools of choice for ingesting data into Hadoop, but their development has slowed and more usable tools are now available. Flume’s configuration file and Sqoop’s command line pale beside modern tools for defining data flow pipelines. While these tools are great for smoothing out impedance mismatches and database […]
The big data world was shocked last week when Apache Hadoop™ vendors Cloudera and Hortonworks announced they would be merging. Anyone familiar with this space knows that these two vendors have hardly been friends, so suffice to say this announcement caught everyone by surprise. Much has already been documented about this merger and many are […]
StreamSets is excited to announce the immediate availability of Control Hub version 3.4.0 and StreamSets Data Collector version 3.5.0. StreamSets Data Collector is the platform’s open source execution engine, moving data between any source and destination, performing transformations and push down analytics along the way. StreamSets Control Hub offers a hosted environment for collaborative design […]
Partners are an important part of driving our business here at StreamSets. The StreamSets DataOps Platform is adopted by leading companies across industries, and in many cases is used directly in conjunction with Cloudera Enterprise. Jointly, customers use StreamSets and Cloudera to build new and innovative solutions that help unlock customer insight, reduce fraud and […]
Minneapolis-based phData has long been a StreamSets partner, deploying the StreamSets DataOps Platform at customers across the US. It’s not surprising then, that when phData principal solutions architect Keith Smith wanted to integrate the Ethereum blockchain platform with the Apache Hadoop filesystem and Apache Kudu, he reached for StreamSets Data Collector.
Join our mailing list to receive the latest news from StreamSets.
