Where change is welcome.
In times of yo-yo-ing markets, rapid change, and economic recession, real time analytics—and the streaming data that makes it possible—can make the difference between barely hanging on and thriving. 2020 has been a tough year, with a global pandemic, a…
StreamSets Data Collector’s HTTP pipeline stages allow a wide range of API integrations. I recently built a data pipeline to integrate customer data from a MySQL database: retrieving, creating and updating leads in Marketo via their REST API. I’ll share…
StreamSets Data Collector is available in the major cloud marketplaces; as an AWS Marketplace vendor, StreamSets has access to a number of data sets detailing product usage and billing. Data sets are available interactively via a web portal and via…
For those of you who joined us on June 6th, you dialed into a forward-thinking conversation between three industry experts. They waxed poetic about topics including big data, DataOps, governance, data science, and more in an effort to help modern…
Data warehouses are a critical component of modern data architecture in enterprises that leverage massive amounts of data to drive quality of their products and services. A data warehouse is an OLAP (Online Analytical Processing) database that collects data from…
StreamSets is pleased to announce a new partnership with Oracle Cloud Infrastructure (OCI). As enterprises move their big data workloads to the cloud, it becomes imperative that their approach to data integration is more resilient and adaptive to continue to serve the business’s needs. This is why StreamSets Data Collector™ is now easily deployable on OCI for adaptive data integration and operations.
What led us to this point? There are fundamental questions such as ‘What good is an Enterprise Data Hub (EDH) without the most current data?’ ‘What good is the EDH without lots of data sources feeding it?’ which leads to the follow up questions of ‘How do you manage data engineering as quickly as software development in a fast-paced DevOps world?’ ‘How do you manage change-data-capture (CDC) from Oracle, streaming log files, and batch SFTP dumps without using large and confusing toolsets?’
To answer all of these questions, StreamSets has created the first complete DataOps (DevOps for data integration) platform to compliment the fail-fast world of DevOps toolsets that are commonly found in places like a cloud-based EDH deployment. Running StreamSets in the Oracle Cloud to support a Cloudera Enterprise Data Hub (EDH) provides an excellent example of DevOps being applied to data to harness the value of a big data project.
Jowanza Joseph is a principal software engineer at One Click Retail with long experience of building reliable and performant distributed data systems. Recently, Jowanza built a pair of data pipelines with StreamSets Data Collector to read data from Ford GoBike and send…
In healthcare, data is delivering life-saving results with predictive capabilities that can address preventable outcomes. The intelligence guiding these initiatives relies on timely data delivery to applications and reviewers. This may involve complex, high velocity data forms with the expectation…
Angel Alvarado is a senior software engineer at One Degree, a San Francisco-based non-profit, and also helps run the Molanco data engineering community. Angel previously contributed a Fun Example of Streaming Data into Minecraft; this time he get serious with the Google Analytics API. Many…
The Spark Evaluator, introduced in StreamSets Data Collector (SDC) version 2.2.0.0, lets you run an Apache Spark application, termed a Spark Transformer, as part of an SDC pipeline. Back in December, we released a tutorial walking you through the process of building a Transformer in Java. Since then, Maurin Lenglart, of Cuberon Labs, has contributed skeleton code for a Scala Transformer, paving the way for a new tutorial, Creating a StreamSets Spark Transformer in Scala.