The DataOps Blog
Where Change Is Welcome
StreamSets Live: Demos with Dash!
45 minutes. Live demos. No (0) slides! Key takeaways: Visually build data pipelines like you code: build, verify, iterate, commit, rinse, repeat! Ingest data from…
Ingest Salesforce Data Into Amazon S3 Data Lake
In this blog, you will learn how to ingest Salesforce data using Bulk API (optimized to process large sets of data) and store it in Amazon Simple Storage Service (Amazon S3) Data Lake using StreamSets Data Collector, a fast data ingestion engine. The primary AWS service used in our data pipeline is Amazon S3, which provides cost effective storage and…
StreamSets Completes SOC 2 Type II Certification for Control Hub
Delivering continuous data in the face of constant change, quickly and confidently, is the StreamSets promise. Part of that confidence is the built-in security we’ve always provided in the technology and processes we follow. Today, we’re proud to announce that we’ve completed SOC 2 Type II certification for StreamSets Control Hub, the first solution to let data professionals operationalize (design,…
Real Time Analytics Is a 2021 Must Have: Here’s How Streaming Data Pipelines to AWS Gets You There
In times of yo-yo-ing markets, rapid change, and economic recession, real time analytics—and the streaming data that makes it possible—can make the difference between barely hanging on and thriving. 2020 has been a tough year, with a global pandemic, a highly contentious US election, and a major recession in the mix. After 100,000+ business closures, hundreds of thousands of job…
Accelerate your DataOps Journey with StreamSets and Intel
This post was co-written by Kirit Basu, VP of Strategy StreamSets, and Brien Porter, Solutions Lead, Intel Corporation. The need for fast access to data has only increased in these turbulent times. To deliver data continuously in the face of constant change and meet exponential growth in demand for data, data-savvy organizations are adopting a DataOps practice. DataOps operationalizes the…
Demystifying Kerberos Authentication on Hadoop Clusters
Guest post by Rishi Jain, Technical Support Engineer III, StreamSets. In this blog post, you'll learn the recommended way of enabling and using kerberos authentication when running StreamSets Transformer, a modern transformation engine, on Hadoop clusters. Generally speaking, the --proxy-user argument to spark-submit allows you to run a Spark job as a different user, besides the one whose keytab you have.…