Where Change Is Welcome
In this blog, you will learn how to design AWS Data Pipelines in StreamSets Data Collector, a fast data ingestion engine, that you can then use to impress your boss with in 2021 -- the year of the data engineer!…
Times are definitely not “normal”, and more challenging for some than others, but I hope everyone is continuing to stay safe and I wish everyone a very happy holiday season. More importantly, I also want to take this opportunity and…
In this post, we will take a look at best practices for integrating StreamSets Data Collector (SDC), a fast data ingestion engine, with Kafka, and take a deep dive into the details with regards to sending Kafka messages to Amazon…
In this blog, you will learn how to ingest changed data from PostgreSQL WAL data and store it in Amazon Simple Storage Service (Amazon S3) using StreamSets Data Collector, a fast data ingestion engine. Consider the use case where you…
Migrating from an on-premises data lake to a cloud data lake does not have to take months or even weeks. With intent-driven data pipelines, you can migrate your data from Hadoop to a cloud data lake in hours. In this…
"Nothing is more real than nothing" - Samuel Beckett While working on an issue for a prospect’s proof of concept pipeline using StreamSets Transformer, a modern Spark ETL engine, I came across a perplexing bug. A certain column in the…
DNB is Norway's largest financial services group, and has a reputation as a trusted financial institution throughout the region. In this guest post, the DNB Data Engineering Centre of Practice team--Saleem Pothiwala, Operations Lead - Customer Insights, Jones Mabea Agwata,…
In this blog, you will learn how to ingest Salesforce data using Bulk API (optimized to process large sets of data) and store it in Amazon Simple Storage Service (Amazon S3) Data Lake using StreamSets Data Collector, a fast data…
Guest post by Rishi Jain, Technical Support Engineer III, StreamSets. In this blog post, you'll learn the recommended way of enabling and using kerberos authentication when running StreamSets Transformer, a modern transformation engine, on Hadoop clusters. Generally speaking, the --proxy-user argument…
This post describes how StreamSets Data Collector, a fast and flexible data ingestion engine, can be configured to take advantage of IAM Roles when running on AWS Elastic Kubernetes Service (EKS). IAM Roles allow data engineers to concentrate on business…