StreamSets Brings DataOps to Hybrid Cloud Architectures
New Features Include Continuous Integration and Deployment (CI/CD), Policy-Driven Data Protection, and Data Drift Handling for Cloud Data
These features build on StreamSets DataOps Platform’s rich catalog of cloud connectors, its cloud-native architecture for easy cross-platform deployment, and its ability to elastically scale dataflows via Kubernetes. Features such as data drift handling and in-stream data protection are powered by StreamSets’ unique Intelligent Pipelines capability, which inspects and analyzes data in-flow, overcoming the lack of visibility common in traditional data integration and big data ingestion approaches.
A majority of StreamSets customers already use the StreamSets DataOps Platform for cloud dataflows, executing both “lift and shift” cloud migration projects that require peak throughput, and continuous real-time streaming of data.
“As our customers embark on their hybrid cloud journey, we see first-hand their struggle to orchestrate end-to-end management of data movement across a growing range of on-premises and cloud platforms,” said Arvind Prabhakar, CTO, StreamSets. “Our DataOps platform was architected as cloud-native from the start, allowing us to easily evolve with the market. Cloud drift-handling and CI/CD for dataflows are unique enhancements that help our customers on their journey from traditional to modern data integration based on DataOps.“
The expansion of data architectures into the cloud creates challenges for enterprises that still rely on traditional data integration software or single-purpose big data ingestion tools. Using these methods, pipelines take too long to build and deploy, and often rely on valuable, specialized developers. They are opaque, denying end-to-end visibility into pipeline performance to prevent failures or detect sensitive personal data in the dataflow. Finally, they are rigid, breaking whenever data drift occurs, such as when fields are added or changed or data platforms are upgraded.
With these new features, which began rolling out in late August, StreamSets DataOps Platform now offers:
- Development automation through a full-featured dataflow designer that includes “easy button” connectors for Amazon S3, Elastic MapReduce (EMR) and RedShift; Azure Data Lake Storage, HDInsight and Azure Databricks; Google DataProc and Snowflake
- Elastic scaling of cloud, multi-cloud and reverse hybrid cloud dataflows via Kubernetes
- New data drift handling, which automatically reflects updates to source schema in Amazon Athena, Azure SQL and Google BigQuery cloud data services
- A new CI/CD framework for automating frequent changes to dataflows through iterative design, test, validate and deployment steps
- New central governance of StreamSets Data Protector policies that detect and deal with sensitive data such as PII and PHI
“While the term DataOps itself has not yet become mainstream, it is clear that the associated concepts and technologies are being widely adopted by enterprises as they seek to become more data-driven,” said Matt Aslett, research vice president, 451 Research. “Businesses are moving toward real-time and self-service cloud analytics executed on large volumes of data, and the complexity and rate of change in data architectures and data user requirements demand the agility and automation promised by the continuous processing and integration of data streams.”
Enterprise data leaders interested in seeing StreamSets DataOps Platform in action can visit StreamSets at the Strata Data Conference (booth #935), September 11-13 in New York City.
DataOps is the application of DevOps practices to data management and integration to reduce the cycle time of data analytics, with a focus on automation, collaboration and monitoring. DataOps is essential for a data landscape marked by architectural complexity with accelerating change.
DataOps is characterized by the following core capabilities:
- Cross-platform data integration that enables flexible selection of fit-for-purpose storage and compute platforms
- Data SLAs for continuous monitoring, measurement and enforcement of business standards for data availability, quality and protection
- Continuous integration and delivery (CI/CD) of dataflows for agility (proactive change management)
- Data drift resilience — automated detection and response to unexpected changes to schema and semantics (reactive change management)
StreamSets built the industry’s first multi-cloud DataOps platform for modern data integration, helping enterprises to continuously flow big, streaming and traditional data to their data scientists and data-intensive applications. It uniquely handles data drift, those frequent and unexpected changes to data that break pipelines and damage data integrity. The platform combines the open source StreamSets Data Collector for execution of any-to-any pipelines (the data plane) with a cloud-native StreamSets Control Hub™ for the design, monitoring and performance management of multi-pipeline topologies (the control plane). Founded in 2014 by Girish Pancha, former chief product officer of Informatica, and Arvind Prabhakar, a former engineering leader at Informatica and Cloudera, StreamSets is backed by top-tier Silicon Valley venture capital firms, including Battery Ventures, New Enterprise Associates (NEA), and Accel Partners. For more information, visit www.streamsets.com.