Where Change Is Welcome
StreamSets DataOps platform provides an end-to-end enterprise solution to maximize the value of your Snowflake Data Cloud. The platform can ingest data into Snowflake (using batch, streaming and change data capture data pipelines). With the preview of the StreamSets Engine…
StreamSets is excited to announce the immediate availability of StreamSets Transformer Engine 4.0.0. It is a modern ETL engine that enables developers and data engineers to build data pipelines and transformations that execute on Apache Spark. Highlights This is our…
Imagine asking Amazon Alexa or Google Home to run your ETL, data processing, and machine learning data pipelines. For example, "Start my data pipeline on Amazon EMR", “How many active jobs do I have running on Databricks?", or "Stop my…
Change Data Capture is becoming essential to migrating to the cloud. In this blog, I have outlined detailed explanations and steps to load Change Data Capture (CDC) data from PostgreSQL to Redshift using StreamSets Data Collector, a fast data ingestion…
Transformer for Snowflake is the first enterprise data transformation engine built on Snowpark. Want to learn how the engine makes advanced, native data transformations for your Data Cloud possible? Join our technical experts on Office Hours. Learn how to load…
Load Data Into Google BigQuery and AutoML In this blog, we will review ETL data pipeline in StreamSets Transformer, a Spark ETL engine, to ingest real-world data from Fire Department of New York (FDNY) stored in Google Cloud Storage (GCS),…
Times are definitely not “normal”, and more challenging for some than others, but I hope everyone is continuing to stay safe and I wish everyone a very happy holiday season. More importantly, I also want to take this opportunity and…
DNB is Norway's largest financial services group, and has a reputation as a trusted financial institution throughout the region. In this guest post, the DNB Data Engineering Centre of Practice team--Saleem Pothiwala, Operations Lead - Customer Insights, Jones Mabea Agwata,…
In this blog, you will learn how to ingest Salesforce data using Bulk API (optimized to process large sets of data) and store it in Amazon Simple Storage Service (Amazon S3) Data Lake using StreamSets Data Collector, a fast data…
Guest post by Rishi Jain, Technical Support Engineer III, StreamSets. In this blog post, you'll learn the recommended way of enabling and using kerberos authentication when running StreamSets Transformer, a modern data transformation engine, on Hadoop clusters. Generally speaking, the --proxy-user…