The convergence of data and application integration is here

Over the past 20 years the relationship between hardware, software and data has changed dramatically.

The consequences for your business have too. The simplicity of hardware, software, and a tiny sliver of data in the middle, contained within a single system in a large box, is unrecognizable by today’s paradigm.

Those changes began around 2000, as we entered the era of VMWare, hypervisors, and early cloud. Enterprises started focusing on software and the role of hardware started to shrink. While this gave rise to DevOps, software and data still existed together as a single unit.

In the past five years, what started as an evolutionary change took a huge step. Today hardware is no longer at the forefront of any conversations. And the handful of big blobs of software that enterprises invested in? They have been atomized into hundreds of smaller, highly complex, agile applications, systems and even microservices in the cloud. This splintered application landscape recharged the integration market and drove massive adoption of iPaaS – Integration Platform-as-a-Service – to easily reconnect key workflows and processes.

The world has changed

But data! Data became the connective tissue ensuring hundreds of otherwise fragmented systems work together as an intelligent whole supporting various business processes. And DataOps—the set of practices and technologies to operationalize data management and integration—came onto the scene blazing.

One way to think about this new paradigm is that instead of sending five jumbo jets lumbering along pre-determined flightpaths, we now have fleets of 500 drones. These can move in any direction freely, reorganize to act as one, split into groups as needed, and address any obstacles—anticipated or not—in their way. And all on-the-fly.

That is, if you can connect them. Can you?

In the new world, StreamSets gets the data to applications, connected and orchestrated by webMethods.io, across complex data pipelines.

Data drifting and the single experience

Part of this new reality is that all these data, and all these systems, are continuously moving in unpredictable ways. “Data drift” is when these unexpected and undocumented changes break processes and corrupt data. It can be as simple as the transition from a 10-digit to 12-digit ID cascading through tens of thousands of applications, or a changed IP address format disrupting data going to a dashboard and going undetected for weeks or months.

But data drift can also reveal new opportunities. Like drifting with data to make sure nothing breaks on either end—so you can speed up innovation. That’s what StreamSets does.

StreamSets, like webMethods, isn’t hardwired into the systems it connects. So whether data needs to be in Amazon Redshift, Snowflake, Google BigQuery or any other system, StreamSets can infer schemas dynamically at runtime and adapt without breaking as a schema changes. Such smart data pipelines are the self-healing connective tissue that gets data where it is needed. .

But this is only half the story. Back in the real world there are so many different patterns in data integration—from batch, to streaming, to CDC and beyond—that a patchwork landscape of technologies has led to huge fragmentation. Data engineers without the right tools end up constantly pivoting to keep things in sight and steady, which is a drag on resources and innovation. StreamSets brings a comprehensive approach to DataOps that addresses all these patterns.

This is still only half the story: Interpreting and taking action on your data is where the rubber meets the road, so to speak. This is where application integration steps in with event-driven, automated workflows and orchestration to transform and activate the data. Adding a powerful integration platform enables your data to provide better visibility and insights, real-time engagement with customers, optimized processes, and frictionless partner and supplier transactions.

webMethods + StreamSets = Better integration, faster innovation

In the case of webMethods and StreamSets, 1 + 1 really is greater than 2. The fragmentation of data and apps—at enterprise scale and with advanced security—is finally being addressed by a single vendor in the hybrid integration space covering data, apps, B2B and APIs.

A good use case for this power is a business using IoT with a continuously flowing, complex maelstrom of data coming in from its devices. In many cases, those devices aren’t sending single data points, but eddying streams consisting of temp, runtime, rotational speed, vibrational delta tolerances, and many more complex machinery calculations.

StreamSets opens continuous data pipelines for seamlessly ingesting and analyzing the data in a cloud data warehouse. The intelligence and insights from the analysis can be leveraged to update applications for device maintenance, technician dispatch, automated resets and more with webMethods.io driving the business logic to close the loop.

So when something is about to fail, you already know. Replacements have been ordered, maintenance already scheduled, suppliers notified, and data captured for a cascade of analysis to update everything from products to processes—automatically without any downtime at all.

Welcome to the convergence

Applications, APIs, data and microservices are the digital backbone that make it possible to run the enterprise. But that backbone needs smart data pipelines—the connective tissue—to let all those independent systems work together and keep data-in-motion safe, secure, resilient, and most importantly, actionable.

For the first time, all technologies are unified on a single platform. That’s the end of piecemeal solutions from multiple vendors: Instead, the combination of data integration, application integration, B2B integration and API integration is a complete hybrid integration platform needed as the market shifts to the cloud.

So, if you want to gain new business insights from data and continuously drive what you’ve learned back to applications and APIs to innovate faster—an even stronger “we” are here to help.

Written in cooperation with Arvind Prabhakar, CTO at StreamSets & Raji Narayana, VP of Product at StreamSets.