Finance teams don’t just keep the lights on in the modern enterprise. Their planning and analysis work is critical to helping business leaders make informed decisions about future strategies. Business agility starts with finance. But making sure the right team members get access to the right data at the right time is not easy. And as data sources multiply, so do the challenges associated with building data pipelines.
New StreamSets research reveals that fixing these chaotic data ecosystems may cost as much as $1.9m per year. But there’s a potentially even greater cost to the business if the finance team’s productivity is impaired.
Why Pipelines Are Breaking
Data pipelines are designed to automatically ingest the most current corporate data and serve it on demand, wherever needed. That’s the kind of confidence line of business (LoB) finance users need to support the organization. But modern data ecosystems are a complex blend of legacy systems, point solutions, custom-built tooling, and cloud-based systems from various suppliers. The result is that data professionals must toil to deliver labor-intensive, one-off solutions. Breakages are frustratingly common.
The vast majority (88%) of data leaders and practitioners working in finance teams say they experience breaks at least once a year, with 42% saying their pipelines break every week. Worryingly, 18% admit incidents occur at least once a day. The most common causes of breakages are:
- Infrastructure changes like migration to a new cloud (44%)
- Bugs and errors introduced during a change (38%)
- Credential changing or expiring (34%)
- Pipelines being too resource intensive (29%)
- Dependencies being removed without knowledge of the data team (27%)
The Bigger Picture
The status quo outlined above is unsustainable. For example, most (79%) finance data professionals claim data in legacy systems, like on-premises databases, is hard for cloud analytics tools to access. Because of the difficulty, they often “don’t bother” to include it when building their pipelines — a serious oversight given the strategic importance of financial, customer, transaction, and tax data residing in legacy systems.
Without resilient data pipelines providing access to all relevant sources of enterprise financial data, organizations will be at risk of:
- Delivering inaccurate financial statements
- Making poor business decisions based on incorrect/incomplete information
- Missing potential financial risks
It doesn’t have to be like this. By centralizing efforts on a single management platform, organizations can finally regain control over the data “wild west” that prevails in too many organizations and unlock data from multiple silos. By building governance into pipelines, they can even empower LoB teams in finance and elsewhere to do last-mile data collection and analysis themselves. That will help drive productivity and digital transformation while removing the IT bottleneck and freeing data professionals to work on more valuable tasks.
To learn more, download our report about the data pipeline challenges facing finance teams — and how to fix them.