Where change is welcome.
The big data world was shocked last week when Apache Hadoop™ data vendors Cloudera and Hortonworks announced they would be merging. Anyone familiar with this space knows that these two vendors have hardly been friends, so suffice to say this…
Partners are an important part of driving our business here at StreamSets. The StreamSets DataOps Platform is adopted by leading companies across industries, and in many cases is used directly in conjunction with Cloudera Enterprise. Jointly, customers use StreamSets and Cloudera to build new and innovative solutions that help unlock customer insight, reduce fraud and risk, and help reduce operational costs. One such customer, Voya Financial, recently won a Data Impact Award for their solution to better protect its customers data while identifying fraud incidents and proactively addressing them. Joint customers Vodafone (Business Impact) and RingCentral (Protect Your Business) were also nominated as finalists.
As cloud adoption grows, so does the complexity of the data architectures that serve as the backbone for modern enterprise applications and the need to enable data integration for hybrid cloud. This complexity, if not planned for, can cripple any cloud initiative. According to research firm Gartner (subscription required), by 2021, at least 75% of large and global organizations will implement a multi-cloud capable hybrid integration platform, up from less than 25% in 2018. Taking a DataOps approach to methods of data integration can help streamline how data is moved around the business and ensure integration initiatives support the cloud-oriented goals of any organization.
The fact is, our founders started our organization on the foundation of DataOps principles and StreamSets was a DataOps company before the term was even coined in late 2015. oOr founders recognized the serious operational challenges that unstructured, streaming data and hybrid cloud infrastructures would pose to enterprises used to static, batch structured data integration. Since our inception, we’ve been focused on enabling teams to operationalize data movement and we created the StreamSets Data Integration Platform to empower customers to capitalize on a DataOps approach.
Detect, Secure and Govern Sensitive Data Upon Ingestion
StreamSets is excited to announce a new product for protecting data in motion. StreamSets Data Protector, as the name would imply, extends the value of the StreamSets DataOps Platform to enable users to detect, secure and govern sensitive data as it flows around your business. Leveraging StreamSets’ unique “Dataflow Sensors”, Data Protector can spot and act on personally identifiable information (PII) at the point of ingest, further strengthening your ability to meet new and changing regulatory requirements such as the General Data Protection Regulation (GDPR) and Health Insurance Portability and Accountability Act (HIPAA).
Control. We always want it, regularly don’t get it, yet in business it’s a must have to ensure things run as expected. Control is particularly critical when it comes to moving data around your company. Without it, it’s difficult to know where data is coming from, where it’s going and how it’s been manipulated (and by whom!) along the way. At StreamSets, we specialize in helping customers effectively control data and move it around their business, from any source to any destination. Over time, we’ve observed that most organizations lack proper controls to adequately ensure that data pipelines are built, executed and operated in a manner that meets the needs of the application or business process which they support.
When it comes to loading data into Apache Hadoop™, the de facto choice for bulk loads of data from leading relational databases is using Apache Sqoop Commands. After initially entering Apache Incubator status in 2011, it quickly saw wide spread adoption and development, eventually graduating to a Top-Level Project (TLP) in 2012.
In StreamSets Data Collector Engine we now have capabilities that enable SDC to behave in a manner almost identical to Sqoop commands. Now customers can use SDC as a way to modernize Sqoop-like workloads, performing the same load functions while getting the ease of use and flexibility benefits that SDC delivers.
Three months into my journey here at StreamSets and I’ve had a chance to talk with many of our customers and prospects to understand how they are using the open source StreamSets Data Collector (SDC) across a number of different use cases. As it turns out, behind solving technical problems in areas such as cybersecurity, IoT or plain old data lake ingestion lies a treasure trove of value that IT teams realize as part of a typical deployment.
While this is not an exhaustive list, let’s take a quick look at some of the more common benefits our customers call out.