Where change is welcome.
If you were lucky enough to get the gift of replacing your existing security event and incident system (SEIM) this year, then there is a chance your organization has considered building a cyber security data lake. Maybe the current solution…
Vinu Kumar is Chief Technologist at HorizonX, based in Sydney, Australia. Vinu helps businesses in unifying data, focusing on a centralized data architecture. In this guest post, reposted from the original here, he explains how to automate solving data quality using open source…
In this blog post, I will present a step-by-step guide on how to scale Data Collector instances on Azure Kubernetes Service (AKS) using provisioning agents—which help automate upgrading and scaling resources on-demand, without having to stop execution of pipeline jobs.…
In my previous blog post, I demonstrated how to achieve low-latency inference using Databricks ML models in StreamSets. Now let's say you have a dataflow pipeline that is ingesting data, enriching it, performing transformations, and based on certain condition(s), you'd…
Transformer for Snowflake is the first enterprise data transformation engine built on Snowpark. Want to learn how the engine makes advanced, native data transformations for your Data Cloud possible? Join our technical experts on Office Hours.
Mike Fuller, a consultant at Red Pill Analytics, has been working on ingesting data into Snowflake’s cloud data warehouse using StreamSets for Snowflake. In this guest blog post, Mike explains how he was able to replicate an Oracle database to Snowflake using the new functionality, both for initial data loading into Snowflake and with change data capture.
Data warehouses are a critical component of modern data architecture in enterprises that leverage massive amounts of data to drive quality of their products and services. A data warehouse is an OLAP (Online Analytical Processing) database that collects data from…
StreamSets is pleased to announce a new partnership with Oracle Cloud Infrastructure (OCI). As enterprises move their big data workloads to the cloud, it becomes imperative that their approach to data integration is more resilient and adaptive to continue to serve the business’s needs. This is why StreamSets Data Collector™ is now easily deployable on OCI for adaptive data integration and operations.
What led us to this point? There are fundamental questions such as ‘What good is an Enterprise Data Hub (EDH) without the most current data?’ ‘What good is the EDH without lots of data sources feeding it?’ which leads to the follow up questions of ‘How do you manage data engineering as quickly as software development in a fast-paced DevOps world?’ ‘How do you manage change-data-capture (CDC) from Oracle, streaming log files, and batch SFTP dumps without using large and confusing toolsets?’
To answer all of these questions, StreamSets has created the first complete DataOps (DevOps for data integration) platform to compliment the fail-fast world of DevOps toolsets that are commonly found in places like a cloud-based EDH deployment. Running StreamSets in the Oracle Cloud to support a Cloudera Enterprise Data Hub (EDH) provides an excellent example of DevOps being applied to data to harness the value of a big data project.
Partners are an important part of driving our business here at StreamSets. The StreamSets DataOps Platform is adopted by leading companies across industries, and in many cases is used directly in conjunction with Cloudera Enterprise. Jointly, customers use StreamSets and Cloudera to build new and innovative solutions that help unlock customer insight, reduce fraud and risk, and help reduce operational costs. One such customer, Voya Financial, recently won a Data Impact Award for their solution to better protect its customers data while identifying fraud incidents and proactively addressing them. Joint customers Vodafone (Business Impact) and RingCentral (Protect Your Business) were also nominated as finalists.
Minneapolis-based phData has long been a StreamSets partner, deploying the StreamSets DataOps Platform at customers across the US. It’s not surprising then, that when phData principal solutions architect Keith Smith wanted to integrate the Ethereum blockchain platform with the Apache Hadoop filesystem and Apache Kudu, he reached for StreamSets Data Collector.
StreamSets has a rich tradition of partnering with Cloudera to highlight companies that are pushing the possible with data and advanced analytics. The Data Impact Awards is an annual event that recognizes the best organizations with data at the center…