Ventana Research Analyst Perspectives

Success with streaming data and events requires a more holistic approach to managing and governing data in motion and data at rest. The use of streaming data and event processing has been part of the data landscape for many decades. For much of that time, data streaming was a niche activity, however, with standalone data streaming and event-processing projects run in parallel with existing batch-processing initiatives, utilizing operational and analytic data platforms. I noted that there has been an increased focus on unified approaches that enable the holistic management and governance of data in motion alongside data at rest. One example is the recent emergence of streaming databases designed to combine the incremental processing capabilities of stream-processing engines with the SQL-based analysis and persistence capabilities of traditional databases.

I have previously written about the importance of data democratization as a key element of a data-driven agenda. Removing barriers that prevent or delay users from gaining access to data enables it to be treated as a product that is generated and consumed, either internally by employees or externally by partners and customers. This is particularly important for organizations adopting the data mesh approach to data ownership, access and governance. Data mesh is an organizational and cultural approach to data, rather than a technology platform. Nevertheless, multiple vendors are increasingly focused on providing products that facilitate adoption of data mesh and promote data democratization. Amazon Web Services is one such vendor, thanks to the recent launch of Amazon DataZone, one of the figurehead analytics and data announcements made during the company’s recent re:Invent customer event.

I recently wrote about the potential use cases for distributed SQL databases as well as techniques being employed by vendors to accelerate adoption. Distributed SQL is a term that is used by several vendors to describe operational data platform products that combine the benefits of the relational database model and native support for distributed cloud architecture, including resilience that spans multiple data centers and/or cloud regions. I noted that compatibility with existing database tools and skills was a key factor for these vendors as they lower barriers to developer adoption. A prime example is Cockroach Labs, which highlighted the importance of compatibility and developer efficiency with the recent launch of CockroachDB 22.2.

I’ve previously written about the analytics continuum, which spans a range of capabilities including reporting, visualization, planning, real-time processes, natural language processing, artificial intelligence and machine learning. I’ve also written about the analysis that goes into making intelligent decisions with decision intelligence. In this perspective, I’d like to focus on one end of the analytics continuum, which I’ll label advanced analytics.

Data observability was a hot topic in 2022 and looks likely to be a continued area of focus for innovation in 2023 and beyond. As I have previously described, data observability software is designed to automate the monitoring of data platforms and data pipelines, as well as the detection and remediation of data quality and data reliability issues. There has been a Cambrian explosion of data observability software vendors in recent years, and while they have fundamental capabilities in common, there is also room for differentiation. One such vendor is Soda Data, which offers an open-source platform for self-service data observability that is focused on facilitating collaboration between business decision-makers and data teams responsible for generating and managing data to improve trust in data.

I have written about the increased demand for data-intensive operational applications infused with the results of analytic processes, such as personalization and artificial intelligence-driven recommendations. I previously described the use of hybrid data processing to enable analytics on application data within operational data platforms. As is often the case in the data platforms sector, however, there is more than one way to peel an orange. Recent years have also seen the emergence of several analytic data platforms that deliver real-time analytic processing suitable for data-intensive operational applications.

I am happy to share insights from our latest Ventana Research Value Index research, which assesses how well vendors’ offerings meet buyers’ requirements. The 2023 Analytic Data Platforms Value Index is the distillation of a year of market and product research by Ventana Research. Drawing on our Benchmark Research, we apply a structured methodology built on evaluation categories that reflect real-world criteria incorporated in a request for proposal to data platform vendors supporting the spectrum of analytic use-cases. Using this methodology, we evaluated vendor submissions in seven categories: five relevant to the Product Experience: Adaptability, Capability, Manageability, Reliability and Usability, and two related to the Customer Experience: Total Cost of Ownership/Return on Investment and Validation. This research-based index evaluates the full business and information technology value of analytic data platforms offerings. I encourage you to learn more about our Value Index and its effectiveness as a vendor selection and request for information/requestion for proposal tool.

Ventana Research recently published the 2023 Analytic Data Platforms Value Index. As organizations strive to be more data-driven, increasing reliance on data as a fundamental factor in business decision-making, the importance of the analytic data platform has never been greater. In this post, I’ll share some of my observations about how the analytic data platforms market is evolving.

I am happy to share insights from our latest Ventana Research Value Index research, which assesses how well vendors’ offerings meet buyers’ requirements. The 2023 Operational Data Platforms Value Index is the distillation of a year of market and product research by Ventana Research. Drawing on our Benchmark Research, we apply a structured methodology built on evaluation categories that reflect real-world criteria incorporated in a request for proposal to data platform vendors supporting the spectrum of operational use cases. Using this methodology, we evaluated vendor submissions in seven categories: five relevant to the Product Experience: Adaptability, Capability, Manageability, Reliability and Usability, and two related to the Customer Experience: Total Cost of Ownership/Return on Investment and Validation.

Ventana Research recently announced its 2023 Market Agenda for Data, continuing the guidance we have offered for two decades to help organizations derive optimal value and improve business outcomes.

For far too long, business intelligence technologies have left the rest of the exercise to the reader. Many of these tools do an excellent job providing information in an interactive way that lets organizations dive into the data and learn a lot about what has happened across all aspects of the business. More recently, many of these tools have added augmented intelligence capabilities that help explain why things happened. But rarely did any of these tools provide information about what to do or how to evaluate the alternative ways in which you might respond.

The shift from on-premises server infrastructure to cloud-based and software-as-a-service (SaaS) models has had a profound impact on the data and analytics architecture of many organizations in recent years. More than one-half of participants (59%) in Ventana Research’s Analytics and Data Benchmark research are deploying data and analytics workloads in the cloud, and a further 30% plan to do so. Customer demand for cloud-based consumption models has also had a significant impact on the products and services that are available from data and analytics vendors. Data platform providers, both operational and analytic, have had to adapt to changing customer demand. The initial response — making existing products available for deployment on cloud infrastructure — only scratched the surface in terms of responding to emerging expectations. We now see the next generation of products, designed specifically to deliver innovation by taking advantage of cloud-native architecture, being brought to market both by emerging startups, and established vendors, including InterSystems.

Analytics processes are all about how organizations use data to create metrics that help manage and improve operations. Yet, the discipline applied to analytics processes seems to be lacking compared to data processes. I’ve pointed out that the weak link in data governance is often analytics. Organizations can also do a better job tying AnalyticOps to DataOps and do more to define and manage metrics. Our research has shown that creating and managing metrics in a semantic model improves analytics processes.

Ventana Research uses the term “data pantry” to describe a method of data storage (and the technology and process blueprint for its construction) created for a specific set of users and use cases in business-focused software. It’s a pantry because all the data one needs is readily available and easily accessible, with labels that are immediately recognized and understood by the users of the application. In tech speak, this means the semantic layer is optimized for the intended audience. It is stocked with data gathered from multiple sources and immediately available for analysis, forecasting, planning and reporting. This does away with the need for analysts to repeatedly perform data extraction, enrichment or transformation motions from the required source systems, all but eliminating the substantial amount of time analysts and business users routinely spend on data preparation.

In previous perspectives in this series, I’ve discussed some of the realities of cloud computing including costs, hybrid and multi-cloud configurations and business continuity. This perspective examines the realities of security and regulatory concerns associated with cloud computing. These issues are often cited by our research participants as reasons they are not embracing the cloud. To be fair, the majority of our research participants are embracing the cloud. However, among those that have not yet made the transition to the cloud, security and regulatory concerns are among the most common issues cited across the various studies we have conducted.

Recently, I suggested you need to “mind the gap” between data and analytics. This perspective addresses another gap — the gap in skills between business intelligence (BI) and artificial intelligence/machine learning (AI/ML).

If you’ve ever been to London, you are probably familiar with the announcements on the London Underground to “mind the gap” between the trains and the platform. I suggest we also need to mind the gap between data and analytics. These worlds are often disconnected in organizations and, as a result, it limits their effectiveness and agility.

I have written a few times in recent months about vendors offering functionality that addresses data orchestration. This is a concept that has been growing in popularity in the past five years amid the rise of Data Operations (DataOps), which describes more agile approaches to data integration and data management. In a nutshell, data orchestration is the process of combining data from multiple operational data sources and preparing and transforming it for analysis. To those unfamiliar with the term, this may sound very much like the tasks that data management practitioners having been undertaking for decades. As such, it is fair to ask what separates data orchestration from traditional approaches to data management. Is it really something new that can deliver innovation and business value, or just the rebranding of existing practices designed to drive demand for products and services?

I have recently written about the organizational and cultural aspects of being data-driven, and the potential advantages data-driven organizations stand to gain by responding faster to worker and customer demands for more innovative, data-rich applications and personalized experiences. I have also explained that data-driven processes require more agile, continuous data processing, with an increased focus on extract, load and transform processes — as well as change data capture and automation and orchestration — as part of a DataOps approach to data management. Safeguarding the health of data pipelines is fundamental to ensuring data is integrated and processed in the sequence required to generate business intelligence. The significance of these data pipelines to delivering data-driven business strategies has led to the emergence of vendors, such as Astronomer, focused on enabling organizations to orchestrate data engineering pipelines and workflows.

Business intelligence has evolved. It now includes a spectrum of analytics, one of the most promising of which has been described as augmented intelligence. Some organizations have used the term to describe the practical reality that artificial intelligence with machine learning is not replacing human intelligence, but augmenting it. The term also represents the application of AI/ML to make business intelligence and analytics tools more powerful and easier to use. It’s this latter usage that I prefer and I’d like to explore in this perspective.

I have written recently about increased demand for data-intensive applications infused with the results of analytic processes, such as personalization and artificial intelligence (AI)-driven recommendations. Almost one-quarter of respondents (22%) to Ventana Research’s Analytics and Data Benchmark Research are currently analyzing data in real time, with an additional 10% analyzing data every hour. There are multiple data platform approaches to delivering real-time data processing and analytics and more agile data pipelines. These include the use of streaming and event data processing, as well as the use of hybrid data processing to enable analytics to be performed on application data within operational data platforms. Another approach, favored by a group of emerging vendors such as Rockset, is to develop these data-intensive applications on a specialist, real-time analytic data platform specifically designed to meet the performance and agility requirements of data-intensive applications.

When I looked at the state of analytics recently, it was clear that analytics are not as widely deployed within organizations as they should be. Only 23% of participants in our Analytics and Data Benchmark Research reported that more than one-half of their organization’s workforce are using analytics. There are many elements to becoming a data-driven organization, as my colleague Matt Aslett points out, but analytics are a necessary component. Our research shows that organizations recognize the importance of embedded analytics, ranking it the second most important digital technology in their analytics and data efforts behind big data and ahead of artificial intelligence and machine learning (AI/ML).

I recently noted that as demand for real-time interactive applications becomes more pervasive, the use of streaming data is becoming more mainstream. Streaming data and event processing has been part of the data landscape for many decades, but for much of that time, data streaming was a niche activity. Although adopted in industry segments with high-performance, real-time data processing and analytics requirements such as financial services and telecommunications, data streaming was far less common elsewhere. That has changed significantly in recent years, fueled by the proliferation of open-source and cloud-based streaming data and event technologies that have lowered the cost and technical barriers to developing new applications able to take advantage of data in-motion. This is a trend we expect to continue, to the extent that streaming data and event processing becomes an integral part of mainstream data-processing architectures.

The analytics and business intelligence market landscape continues to grow as more organizations seek robust tools and capabilities to visualize and better understand data. BI systems are used to perform data analysis, identify market trends and opportunities and streamline business processes. They can collect and combine data from internal and external systems to present a holistic view.

I often use the term “analytics” to refer to a broad set of capabilities, deliberately broader than business intelligence. In this Perspective, I’d like to share what decision-makers should consider as they evaluate the range of analytics requirements for their organization.

When joining Ventana Research, I noted that the need to be more data-driven has become a mantra among large and small organizations alike. Data-driven organizations stand to gain competitive advantage, responding faster to worker and customer demands for more innovative, data-rich applications and personalized experiences. Being data-driven is clearly something to aspire to. However, it is also a somewhat vague concept without clear definition. We know data-driven organizations when we see them — the likes of Airbnb, DoorDash, ING Bank, Netflix, Spotify, and Uber are often cited as examples — but it is not necessarily clear what separates the data-driven from the rest. Data has been used in decision-making processes for thousands of years, and no business operates without some form of data processing and analytics. As such, although many organizations may aspire to be more data-driven, identifying and defining the steps required to achieve that goal are not necessarily easy. In this Analyst Perspective, I will outline the four key traits that I believe are required for a company to be considered data-driven.

We’ve recently published our latest Benchmark Research on Data Governance and it’s fair to say, “you’ve come a long way, baby.” Many of you reading this weren’t around when that phrase was introduced in 1968 to promote Virginia Slims cigarettes, but you may have heard the phrase because it went on to become a part of popular culture. We’ve learned a lot about cigarettes since then, and we’ve learned a lot about data governance, too.

I recently wrote about the growing range of use cases for which NoSQL databases can be considered, given increased breadth and depth of functionality available from providers of the various non-relational data platforms. As I noted, one category of NoSQL databases — graph databases — are inherently suitable for use cases that rely on relationships, such as social media, fraud detection and recommendation engines, since the graph data model represents the entities and values and also the relationships between them. The native representation of relationships can also be significant in surfacing “features” for use in machine learning modeling. There has been a concerted effort in recent years by graph database providers, including TigerGraph, to encourage and facilitate the use of graph databases by data scientists to support the development, testing and deployment of machine learning models.

I previously described the concept of hydroanalytic data platforms, which combine the structured data processing and analytics acceleration capabilities associated with data warehousing with the low-cost and multi-structured data storage advantages of the data lake. One of the key enablers of this approach is interactive SQL query engine functionality, which facilitates the use of existing business intelligence (BI) and data science tools to analyze data in data lakes. Interactive SQL query engines have been in use for several years — many of the capabilities were initially used to accelerate analytics on Hadoop — but have evolved along with data lake initiatives to enable analysis of data in cloud object storage. The open source Presto project is one of the most prominent interactive SQL query engines and has been adopted by some of the largest digital-native organizations. Presto managed-services provider Ahana is on a mission to bring the advantages of Presto to the masses.

I’ve never been a fan of talking about semantic models because most of the workforce probably doesn’t understand what they are, or doesn’t recognize them by name. But the findings in our recent Analytics and Data Benchmark Research have changed my mind. The research shows how important a semantic model can be to the success of data and analytics processes. Organizations that have successfully implemented a semantic model are more than twice as likely to report satisfaction with analytics (77%) compared with a 33% overall satisfaction rate. Therefore, I owe it to all of you to write about them.

As I recently described, it is anticipated that the majority of database workloads will continue to be served by specialist data platforms targeting operational and analytic workloads, albeit with growing demand for hybrid data processing use-cases and functionality. Specialist operational and analytic data platforms have historically been the since preferred option, but there have always been general-purpose databases that could be used for both analytic and operational workloads, with tuning and extensions to meet the specific requirements of each.

I recently described the use cases driving interest in hybrid data processing capabilities that enable analysis of data in an operational data platform without impacting operational application performance or requiring data to be extracted to an external analytic data platform. Hybrid data processing functionality is becoming increasingly attractive to aid the development of intelligent applications infused with personalization and artificial intelligence-driven recommendations. These applications can be used to improve customer service; engagement, detect and prevent fraud; and increase operational efficiency. Several database providers now offer hybrid data processing capabilities to support these application requirements. One of the vendors addressing this opportunity is SingleStore.

There is a fundamental flaw in information technology, or at least in the way it is most commonly delivered. Most technology systems are developed under the assumption that all people will use the system primarily in the same way. Sure, there are some options built in — perhaps the same action can be initiated by either clicking on a button, selecting a menu item or invoking a keyboard short-cut. The problem is that when every variation needs to be coded into the system, the prospect of providing personalized software programs to every individual is impractical.

The server is a key component of enterprise computing, providing the functional compute resources required to support software applications. Historically, the server was so fundamentally important that it – along with the processor, or processor core – was also a definitional unit by which software was measured, priced and sold. That changed with the advent of cloud-based service delivery and consumption models.

Over a decade ago, I coined the term NewSQL to describe the new breed of horizontally scalable, relational database products. The term was adopted by a variety of vendors that sought to combine the transactional consistency of the relational database model with elastic, cloud-native scalability. Many of the early NewSQL vendors struggled to gain traction, however, and were either acquired or ceased operations before they could make an impact in the crowded operational data platforms market. Nonetheless, the potential benefits of data platforms that span both on-premises and cloud resources remain. As I recently noted, many of the new operational database vendors have now adopted the term “distributed SQL” to describe their offerings. In addition to new terminology, a key trend that separates distributed SQL vendors from the NewSQL providers that preceded them is a greater focus on developers, laying the foundation for the next generation of applications that will depend on horizontally scalable, relational-database functionality. Yugabyte is a case in point.

I recently wrote about the importance of data pipelines and the role they play in transporting data between the stages of data processing and analytics. Healthy data pipelines are necessary to ensure data is integrated and processed in the sequence required to generate business intelligence. The concept of the data pipeline is nothing new of course, but it is becoming increasingly important as organizations adapt data management processes to be more data driven.

I recently described the growing level of interest in data mesh which provides an organizational and cultural approach to data ownership, access and governance that facilitates distributed data processing. As I stated in my Analyst Perspective, data mesh is not a product that can be acquired or even a technical architecture that can be built. Adopting the data mesh approach is dependent on people and process change to overcome traditional reliance on centralized ownership of data and infrastructure and adapt to its principles of domain-oriented ownership, data as a product, self-serve data infrastructure and federated governance. Many organizations will need to make technological changes to facilitate adoption of data mesh, however. Starburst Data is associated with accelerating analysis of data in data lakes but is also one of several vendors aligning their products with data mesh.

I have written previously that the world of data and analytics will become more and more centered around real-time, streaming data. Data is created constantly and increasingly is being collected simultaneously. Technology advances now enable organizations to process and analyze information as it is being collected to respond in real time to opportunities and threats. Not all use cases require real-time analysis and response, but many do, including multiple use cases that can improve customer experiences. For example, best-in-class e-commerce interactions should provide real-time updates on inventory status to avoid stock-out or back-order situations. Customer service interactions should provide real-time recommendations that minimize the time to resolution. Location-based offers should be targeted at the customer’s current location, not their location several minutes ago. Another domain where real-time analyses are critical is internet of things (IoT) applications. Additionally, use cases like predictive maintenance require timely information to prevent equipment failures that help avoid additional costs and damage.

Data mesh is the latest trend to grip the data and analytics sector. The term has been rapidly adopted by numerous vendors — as well as a growing number of organizations —as a means of embracing distributed data processing. Understanding and adopting data mesh remains a challenge, however. Data mesh is not a product that can be acquired, or even a technical architecture that can be built. It is an organizational and cultural approach to data ownership, access and governance. Adopting data mesh requires cultural and organizational change. Data mesh promises multiple benefits to organizations that embrace this change, but doing so may be far from easy.

For years, maybe decades, we have heard about the struggles between IT and line-of-business functions. In this perspective, we will look at some of the data from our Analytics and Data Benchmark Research about the roles of IT and line-of-business teams in analytics and data processes. We will also look at some of the disconnects between these two groups. And, by looking at how organizations are operating today and the results they are achieving, we can discern some of the best practices for improving the outcomes of analytics and data processes.

As businesses become more data-driven, they are increasingly dependent on the quality of their data and the reliability of their data pipelines. Making decisions based on data does not guarantee success, especially if the business cannot ensure that the data is accurate and trustworthy. While there is potential value in capturing all data — good or bad — making decisions based on low-quality data may do more harm than good.

I recently described how the data platforms landscape will remain divided between analytic and operational workloads for the foreseeable future. Analytic data platforms are designed to store, manage, process and analyze data, enabling organizations to maximize data to operate with greater efficiency, while operational data platforms are designed to store, manage and process data to support worker-, customer- and partner-facing operational applications. At the same time, however, we see increased demand for intelligent applications infused with the results of analytic processes, such as personalization and artificial intelligence-driven recommendations. The need for real-time interactivity means that these applications cannot be served by traditional processes that rely on the batch extraction, transformation and loading of data from operational data platforms into analytic data platforms for analysis. Instead, they rely on analysis of data in the operational data platform itself via hybrid data processing capabilities to accelerate worker decision-making or improve customer experience.

Ventana Research recently announced its 2022 Market Agenda for Analytics, continuing the guidance we have offered for nearly two decades to help organizations derive optimal value from technology investments in order to improve business outcomes.

Organizations today are working with multiple applications and systems, including enterprise resource planning (ERP), customer relationship management (CRM), supply chain management (SCM) and other systems, where data can easily become fragmented and siloed. And as the organization increases its data sources and adds more systems and custom applications, it becomes challenging to manage the data consistently and keep data definitions up to date. This increases the need to use master data management (MDM) software that can provide a single source of truth to drive accurate analytics and business operations.

The need for data-driven decision-making requires organizations to transform not only the approach to business intelligence and data science but also accelerate the development of new operational applications that support greater business agility, enable cloud- and mobile-based consumption, and deliver more interactive and personalized experiences. To stay competitive, organizations need to prioritize the development of new, data-driven applications. As a result, many have been encouraged to invest in new data platforms designed to support agile development and cloud-based delivery. This is one of the factors driving the growth of MongoDB, and continues to drive the evolution of its document database into what is now described as a cloud-based application data platform.