Ventana Research Analyst Perspectives

Success with streaming data and events requires a more holistic approach to managing and governing data in motion and data at rest. The use of streaming data and event processing has been part of the data landscape for many decades. For much of that time, data streaming was a niche activity, however, with standalone data streaming and event-processing projects run in parallel with existing batch-processing initiatives, utilizing operational and analytic data platforms. I noted that there has been an increased focus on unified approaches that enable the holistic management and governance of data in motion alongside data at rest. One example is the recent emergence of streaming databases designed to combine the incremental processing capabilities of stream-processing engines with the SQL-based analysis and persistence capabilities of traditional databases.

Embedded business intelligence (BI) continues to transform the business landscape, enabling organizations to quickly interpret data and convert it into actionable insights. It allows organizations to extract information in real time and answer wide-ranging business questions. Embedding analytics helps tackle the issue of extracting information from data which is a time-consuming process. Our research shows organizations spend more time cleaning and optimizing data for analysis rather than creating insights. On top of that, they are adding more data sources and information systems which in turn introduces more complexity. Our Analytics and Data Benchmark Research shows that organizations face various challenges with analytics and BI. More than one-third of participants (35%) responded that they find it hard to integrate analytics and BI with business processes and connect to multiple data sources. By embedding analytics and BI into business processes and workflows, organizations can enable users to make critical decisions fast, enhancing overall business agility.

As I recently pointed out, process mining has emerged as a pivotal technology for data-driven organizations to discover, monitor and improve processes through use of real-time event data, transactional data and log files. With recent advancements, process mining has become more efficient at discovering insights in complex processes using algorithms and visualizations. Organizations use it to better understand the current state of systems and business processes. It is also used to enable business process intelligence and improvement in any function or industry using events and activity models for data-driven decision-making. We assert that through 2024, 1 in 4 organizations will look to streamline their operations by exploring process mining to optimize workflow and business processes.

I recently wrote about the need for organizations to take a holistic approach to the management and governance of data in motion alongside data at rest. As adoption of streaming data and event processing increases, it is no longer sufficient for streaming data projects to exist in isolation. Data needs to be managed and governed regardless of whether it is processed in batch or as a stream of events. This requirement has resulted in established data management vendors increasing their focus on streaming data and event processing through product development as well as acquisitions. It has also resulted in streaming and event specialists, such as Confluent, adding centralized management and governance capabilities to their existing offerings as they seek to establish or reinforce the strategic importance of streaming data as part of a modern approach to data management.

I have written recently about increased demand for data-intensive applications infused with the results of analytic processes, such as personalization and artificial intelligence (AI)-driven recommendations. Almost one-quarter of respondents (22%) to Ventana Research’s Analytics and Data Benchmark Research are currently analyzing data in real time, with an additional 10% analyzing data every hour. There are multiple data platform approaches to delivering real-time data processing and analytics and more agile data pipelines. These include the use of streaming and event data processing, as well as the use of hybrid data processing to enable analytics to be performed on application data within operational data platforms. Another approach, favored by a group of emerging vendors such as Rockset, is to develop these data-intensive applications on a specialist, real-time analytic data platform specifically designed to meet the performance and agility requirements of data-intensive applications.

Organizations are managing and analyzing large datasets every day, identifying patterns and generating insights to inform decisions. This can provide numerous benefits for an organization, such as improved operational efficiency, cost optimization, fraud detection, competitive advantage and enhanced business processes. By bringing the right, actionable data to the right user, organizations can potentially speed up processes and make more effective operational decisions.

I recently noted that as demand for real-time interactive applications becomes more pervasive, the use of streaming data is becoming more mainstream. Streaming data and event processing has been part of the data landscape for many decades, but for much of that time, data streaming was a niche activity. Although adopted in industry segments with high-performance, real-time data processing and analytics requirements such as financial services and telecommunications, data streaming was far less common elsewhere. That has changed significantly in recent years, fueled by the proliferation of open-source and cloud-based streaming data and event technologies that have lowered the cost and technical barriers to developing new applications able to take advantage of data in-motion. This is a trend we expect to continue, to the extent that streaming data and event processing becomes an integral part of mainstream data-processing architectures.

I often use the term “analytics” to refer to a broad set of capabilities, deliberately broader than business intelligence. In this Perspective, I’d like to share what decision-makers should consider as they evaluate the range of analytics requirements for their organization.

Streaming data has been part of the industry landscape for decades but has largely been focused on niche applications in segments with the highest real-time data processing and analytics performance requirements, such as financial services and telecommunications. As demand for real-time interactive applications becomes more pervasive, streaming data is becoming a more mainstream pursuit, aided by the proliferation of open-source streaming data and event technologies, which have lowered the cost and technical barriers to developing new applications that take advantage of data in motion. Ventana Research’s Streaming Data Dynamic Insights enables an organization to assess its relative maturity in achieving value from streaming data. I assert that by 2024, more than one-half of all organizations’ standard information architectures will include streaming data and event processing, allowing organizations to be more responsive and provide better customer experiences.

When joining Ventana Research, I noted that the need to be more data-driven has become a mantra among large and small organizations alike. Data-driven organizations stand to gain competitive advantage, responding faster to worker and customer demands for more innovative, data-rich applications and personalized experiences. Being data-driven is clearly something to aspire to. However, it is also a somewhat vague concept without clear definition. We know data-driven organizations when we see them — the likes of Airbnb, DoorDash, ING Bank, Netflix, Spotify, and Uber are often cited as examples — but it is not necessarily clear what separates the data-driven from the rest. Data has been used in decision-making processes for thousands of years, and no business operates without some form of data processing and analytics. As such, although many organizations may aspire to be more data-driven, identifying and defining the steps required to achieve that goal are not necessarily easy. In this Analyst Perspective, I will outline the four key traits that I believe are required for a company to be considered data-driven.

Organizations are continuously increasing the use of analytics and business intelligence to turn data into meaningful and actionable insights. Our Analytics and Data Benchmark Research shows some of the benefits of using analytics: Improved efficiency in business processes, improved communication and gaining a competitive edge in the market top the list. With a unified BI system, organizations can have a comprehensive view of all organizational data to better manage processes and identify opportunities.

Organizations have been using data virtualization to collect and integrate data from various sources, and in different formats, to create a single source of truth without redundancy or overlap, thus improving and accelerating decision-making giving them a competitive advantage in the market. Our research shows that data virtualization is popular in the big data world. One-quarter (27%) of participants in our Data Lake Dynamic Insights Research reported they were currently using data virtualization, and another two-quarters (46%) planned to include data virtualization in the future. Even more interesting, those who are using data virtualization reported higher rates of satisfaction (79%) with their data lake than those who are not (36%). Our Analytics and Data Benchmark Research shows more than one-third of organizations (37%) are using data virtualization in that context. Here, too, those using data virtualization reported higher levels of satisfaction (88%) than those that are not (66%).

I have written previously that the world of data and analytics will become more and more centered around real-time, streaming data. Data is created constantly and increasingly is being collected simultaneously. Technology advances now enable organizations to process and analyze information as it is being collected to respond in real time to opportunities and threats. Not all use cases require real-time analysis and response, but many do, including multiple use cases that can improve customer experiences. For example, best-in-class e-commerce interactions should provide real-time updates on inventory status to avoid stock-out or back-order situations. Customer service interactions should provide real-time recommendations that minimize the time to resolution. Location-based offers should be targeted at the customer’s current location, not their location several minutes ago. Another domain where real-time analyses are critical is internet of things (IoT) applications. Additionally, use cases like predictive maintenance require timely information to prevent equipment failures that help avoid additional costs and damage.

Organizations of all sizes are dealing with exponentially increasing data volume and data sources, which creates challenges such as siloed information, increased technical complexities across various systems and slow reporting of important business metrics. Migrating to the cloud does not solve the problems associated with performing analytics and business intelligence on data stored in disparate systems. Also, the computing power needed to process large volumes of data consists of clusters of servers with hundreds or thousands of nodes that can be difficult to administer. Our Analytics and Data Benchmark Research shows that organizations have concerns about current analytics and BI technology. Findings include difficulty integrating data with other business processes, systems that are not flexible enough to scale operations and trouble accessing data from various data sources.

How does your organization define and display its metrics? I believe many organizations are not defining and displaying metrics in a way that benefits them most. If an organization goes through the trouble of measuring and reporting on a metric, the analysis ought to include all the information needed to evaluate that metric effectively. A number, by itself, does not provide any indication of whether the result is good or bad. Too often, the reader is expected to understand the difference, but why leave this evaluation to chance? Why not be more explicit about what results are expected?

Our research shows that nearly all financial service organizations (97%) consider it important to accelerate the flow of information and improve responsiveness. Even just a few years ago, capturing and evaluating this information quickly was much more challenging, but with the advent of streaming data technologies that capture and process large volumes of data in real time, financial service organizations can quickly turn events into valuable business outcomes in the form of new products and services or revenue.

The work environment today demands that your organization advances the efficiency to execute business processes for continuous operations to have a positive impact on business performance. The capability to be responsive to any range of minor to disruptive business events is required to support business continuity and level of organizational readiness to meet the needs of digital business. Ventana Research asserts that in 2025, one-quarter of organizations will remain digitally ineffective in achieving the business priorities for customer-, product- and people-related processes. It is essential to eliminate bottlenecks and become an organization that places action and decision-making at is center to optimize the execution of business processes.

Access to external data can provide a competitive advantage. Our research shows that more than three-quarters (77%) of participants consider external data to be an important part of their machine learning (ML) efforts. The most important external data source identified is social media, followed by demographic data from data brokers. Organizations also identified government data, market data, environmental data and location data as important external data sources. External data is not just part of ML analyses though. Our research shows that external data sources are also a routine part of data preparation processes, with 80% of organizations incorporating one or more external data sources. And a similar proportion of participants in our research (84%) include external data in their data lakes.

Subscription management and billing services help organizations offer unique benefits and enhance delivery to customers. By making services more personalized, organizations can acquire – and retain – more customers.

Customer support operations increasingly rely on automation and complex workflow processes to reduce costs and improve experiences. Automation also allows organizations to make their service processes richer, incorporating information and staff from back offices, for example, or embedding conversational tools into contact center processes.

Ventana Research recently announced its 2021 market agenda for Analytics, continuing the guidance we’ve offered for nearly two decades to help organizations derive optimal value from technology investments to improve business outcomes.