Taking a broader view on modern iPaaS

The integration of applications, data, devices, and processes is critical in order to stream and store data for instant analysis and solution building.

This just got a lot easier thanks to Software AG’s acquisition of StreamSets, the leader in data integration for the modern data stack.

Data analysis is crucial in today’s market; it helps your company to improve processes, to provide excellent customer service, and win against the competition. Data is everywhere, from your internal applications like SAP or Salesforce or any external services for data that can add context, like traffic or weather.

Today, the lines between application integration and data integration are merging, as companies strive to ingest and analyze vast quantities of information. The problem is that there is so much data – internal, external, structured, and unstructured – that managing it is difficult and time-consuming. Why?

Traditionally, business analytics relied on well-defined data sets used in predictable ways. But today’s data is messy – often semi- or unstructured. Data producers and consumers find it difficult to store and then analyze in the way they are used to.

To get your data to where you want it, in the format you need for analysis, you need several steps. First, you’ll probably decide to use a cloud data lake or warehouse provider to store your data. Then you’ll subscribe to several data sources. But this is just the beginning. Because each data feed has a different structure and type of data. Managing more than one – at the same time – is a complex task.

Managing and operating the flow of data traffic

Think of your cloud database like a large car park: Your data is parked there so you can use it as and when you want. Then, imagine you have 10 data roads leading to it, each with a different speed limit and number of cars traveling on them. Throw in some complex interchanges and junctions, where fast cars need to reduce their speed and wait for the slow cars to join.

This would be like Autobahn traffic in Germany trying to merge with London traffic at rush hour. There would be crashes, bottlenecks would back up the roads with thousands of cars (data). Then imagine the cars on the road are changing into vans or trucks as they travel. Your car park could fill up with cars you can’t drive, or ones you don’t need. You could have a lot full of moving vans, when what you need is a passenger car.

Data pipeline

This is where a data pipeline comes in. You build the road, but the data pipeline is what manages the traffic, so that it arrives on time, intact and is instantly useable. Then, and only then, can you apply analytics, AI or ML to get the real-time decision-making capabilities you need.

To achieve this, you need two things:

• To connect data sources via real-time messaging: Application integration
• To move and transform large amounts of data into data warehouses and data lakes intact and usable: Data integration.

Software AG + StreamSets = flexible, simple solution

For a complete integration platform as a service (iPaaS) solution, the foundation for modern analytics and new business building, there are four main pillars:

Existing Software AG solutions fulfill the first three – application integration and API management with webMethods, IoT device integration with Cumulocity IoT and process integration with ARIS.

StreamSets completes the package by providing the fourth pillar – generic data integration. This combination creates a “mission control” for enterprise integration, with full visibility, preventing data drift – irrespective of data runtime or structure changes. And to expose and monetize the benefits of such an integrated world of apps, devices, and processes, one additional ingredient is absolutely key: An enterprise-wide API management and access capability. 

Mutual benefits for our customers

Process mining. Process mining is analytics, and analytics require data. StreamSets can provide that data in any time, format and volume required.

IoT. Internet of Things customers also need data that does not come from devices, such as contextual and situational data from external sources. StreamSets can provide that data in any timeframe, format and volume required.

Hybrid integration: Where Software AG’s existing digital business portfolio connects a hybrid application landscape and integrates the transaction data within it, StreamSets’ smart data pipeline technology collects, consolidates and moves data from applications, the edge and processes, with an understanding of the structure and meaning of the data passing through them. It can transport the data to a set of hybrid destinations, such as modern cloud data warehouses, data lakes, messaging systems and event hubs.

Conclusion

StreamSets’ smart data integration platform moves data seamlessly and securely to and from any part of an organization’s digital backbone that want to unlock and capture more value from their data. It easily manages the most diverse and heterogenous datasets, reflecting todays and future needs of digitally savvy enterprises.

Software AG is thrilled to welcome the whole StreamSets team, welcoming outstanding colleagues with a track record of innovation and success. I am incredibly excited at the prospect of working with them to grow our business together.