DataOps: Conquering Data in the Age of Modern Analytics
StreamSets’ platform emerges as key technical foundation enabling the DataOps “killer approach” for modernizing how data is managed and integrated
The current state of data
Data is fragmented and proliferating in more places, with more and more of it outside the direct control of the IT department. At the same time, data is more valuable than ever. Analytics has modernized to harness data in an always-on, always-changing world. How data is delivered to drive those analytics has to modernize, too. DataOps — based on the DevOps concepts of continuous delivery and bringing data providers and data consumers closer together — emerges as a new approach to bring data integration and management into the modern era.
As outlined in “DataOps: The Authoritative Edition,” a book written by industry expert John Schmidt and Kirit Basu, VP Product, StreamSets, DataOps is a rapidly emerging practice that is critical for modernizing and operationalizing data management and integration in order to keep pace with the insatiable and accelerating business demands of modern analytics. (Buy the book here.) A DataOps practice enables data professionals to deal with the speed, fragmentation and pace of change associated with modern analytics. Ultimately, by embracing DataOps practices and the technology enabling DataOps, data professionals are no longer forced to choose between rapid delivery of data and the confidence that it’s reliable; rather, they can have both.
Data management industry is taking notice
DataOps’ rise has been documented by a number of industry analysts and experts, including 451 Research and Eckerson Group. In Gartner’s “Hype Cycle for Data Management, 20191” (Gartner subscription required), DataOps is positioned in the Innovation Trigger phase of the Hype Cycle and noted as “On the Rise.” In fact, the Gartner “Your Data Culture Is Changing – Do You Need DataOps?2” presentation states, “Given the tremendous pressure to achieve faster delivery of new and enhanced data and analytics capabilities, DataOps will quickly traverse the first half of the Hype Cycle.”
We believe the real testament to DataOps’ rise, however, is the growing number of leading companies that are building out their DataOps practices and investing in the people, processes and technology to support it. Many of these companies, such as GlaxoSmithKline, RingCentral and Shell, are StreamSets customers using the StreamSets DataOps Platform. This technology bolsters their DataOps practices by addressing data drift — the unpredictable, unannounced and unending changes in data characteristics caused by the operation, maintenance and modernization of the systems that produce the data. The platform mitigates data drift by detecting and handling these changes in an automated way, and enables continuous design and operations.
Many of these StreamSets customers are presenting at the StreamSets DataOps Summit 2019, the first professional conference devoted to thought leadership and technical education for the emerging domain of DataOps. At the DataOps Summit, they will share their technology use cases and DataOps best practices with other StreamSets customers, partners, open source users and industry experts. The event is taking place September 3-5 in San Francisco.
“DataOps is transforming the delivery of data by using a continuous improvement approach to acquiring, aligning, rationalizing and evolving the data to meet the needs of the consumer,” said Mark Ramsey, former Chief Data Officer, GlaxoSmithKline. “StreamSets is a critical technology enabler for that continual flow of data.”
“It’s very early in the game for DataOps, with a number of different organizations and vendors offering their own definition of the term in an effort to be part of the excitement,” said Girish Pancha, CEO, StreamSets. “But the practice is real, as evidenced by our customers who have built teams and processes around it. We are committed to not only innovating on the technology front with our StreamSets DataOps platform, but also supporting the broader ecosystem of practitioners, thought leaders and vendors who are maturing this critical practice for modernizing data management.”
1Gartner Hype Cycle for Data Management, 2019, Donald Feinberg, Adam Ronthal, 31 July 2019
2Gartner Your Data Culture Is Changing — Do You Need DataOps?, Nick Heudecker, et al, 10 June 2019
Gartner does not endorse any vendor, product or service depicted in its research publications, and does not advise technology users to select only those vendors with the highest ratings or other designation. Gartner research publications consist of the opinions of Gartner’s research organization and should not be construed as statements of fact. Gartner disclaims all warranties, expressed or implied, with respect to this research, including any warranties of merchantability or fitness for a particular purpose.
Analytics has modernized in our always-on, always-changing world. How you deliver data to drive analytics has to modernize, too. DataOps is a set of practices and technologies that operationalizes data management and integration to ensure resiliency and agility despite ceaseless change. It combines the DevOps principles of continuous delivery with the ability to tame data drift (unexpected and undocumented changes to data). By embedding these principles, DataOps makes it possible to deliver the continuous data needed to drive modern analytics and digital transformation.
StreamSets built the industry’s first multi-cloud DataOps platform for modern data integration, helping enterprises to continuously flow big, streaming and traditional data to their data science and data analytics applications. The platform uniquely handles data drift, those frequent and unexpected changes to upstream data that break pipelines and damage data integrity. The StreamSets DataOps Platform allows for execution of any-to-any pipelines, ETL processing and machine learning with a cloud-native operations portal for the continuous automation and monitoring of complex multi-pipeline topologies.
Founded in 2014, StreamSets is backed by top-tier Silicon Valley venture capital firms, including Battery Ventures, New Enterprise Associates (NEA), and Accel Partners. For more information, visit www.streamsets.com.
StreamSets and the StreamSets Logo are the registered trademarks of StreamSets, Inc. All other marks referenced are the property of their respective owners.