Ventana Research Analyst Perspectives

I am happy to share some insights from our latest Value Index research, which rates how well vendors’ offerings meet buyers’ performance requirements in seven categories, five relevant to the product (adaptability, capability, manageability, reliability and usability) and two related to the vendor (TCO/ROI and vendor validation). The Ventana Research Value Index on Learning Management Systems 2020 (LMS) is the distillation of a year of market and product research efforts by Ventana Research. Drawing on our benchmark research and expertise, this research-based index is the first such evaluation to assess the full business value of learning management software.

Unlike many IT analyst firms that rank vendors from an IT-only perspective or consider future plans or vision over what is available in the products today, Ventana Research has designed the Value Index to provide a balanced perspective of vendors and products that is rooted in an understanding of business drivers and needs. This approach not only reduces cost and time but also minimizes the risk of making a decision that is bad for the business. Using the Value Index will enable your organization to achieve the levels of efficiency and effectiveness needed to optimize learning management.

This Value Index report evaluates the following vendors that offer products that provide learning management as we define it: Cegid, Cornerstone OnDemand, D2L, Docebo, Infor, Oracle, PeopleFluent, Saba, SAP SuccessFactors, Skillsoft, SumTotal Systems and Workday.

The Value Index for LMS finds Cornerstone OnDemand first on the list with Saba in second place and SAP SuccessFactors in third. Companies that rank in the top three in any category earn the designation Value Index Leader. Cornerstone OnDemand has done so in six of the seven evaluation categories; Oracle, SAP SuccessFactors and Workday are Value Index Leaders in three categories and Saba and Skillsoft are leaders in two.

In this Value Index, the overall scores span a wide range. This reflects the finding of our research that the current set of leading learning management products are at different stages of maturity, with products ranging from three to more than 20 years on the market. Each demonstrates a different level of commitment to — and ongoing investment in — product innovations in the learning management space and offer varying degrees of customer assurance to accompany their product offerings. Additionally, some vendors have invested in making their learning management system more functionally robust but have lagged in modernizing the user experience. In other cases, vendors have prioritized the user experience and provide a range of valuable functionality, but the underlying platform architecture still lags in areas such as product reliability and performance, manageability or adaptability.

The Value Index for LMS, in common with all our Value Index research, uses the Ventana Research Value Index methodology, a framework that evaluates vendors and their products in seven categories. Five are product-related, while two assess customer assurance. The product categories are Capability, Usability, Adaptability, Manageability and Reliability; when we examine performance in these five product categories together, the vendors ranking highest are Value Index Leaders Cornerstone OnDemand, SAP SuccessFactors and Oracle.

We weighted the Capability category at 20% of this Value Index. This relatively significant weighting reflects the emphasis that our research shows learning management system customers place on capabilities in their selection process. This category examined products with respect to how LMS content is found, curated and packaged and the level of sophistication with which the system recommends relevant content to learners. Many vendors are providing “just-in-time” and “in-the-flow-of-work” learning that’s enhanced with social collaboration features. We find that many LMS vendors also have addressed the need for adaptive learning; these systems have the built-in intelligence to seamlessly address learners’ knowledge gaps, often in the context of learning paths or curriculums.

This assessment finds wider variations in Capability category areas such as integration with complementary learning components, including competency models, performance management and other HCM processes, assessments and testing, coaching and immersive (augmented and virtual reality) platforms as well as chat and messaging tools. Additionally, some vendors’ products emphasize career management, evidence of understanding, ease of authoring and content merging tools, and various analytics use cases that demonstrate the impact and adoption of learning management investments. Finally, certain vendors have made more progress than others in micro- or nano-learning, where instruction occurs in short burst, sometimes in 10 minutes or less. In this category Cornerstone OnDemand, SAP SuccessFactors and Saba are Value Index Leaders.

Our Value Index analysis of a product’s Usability evaluates how it addresses a wide range of business needs. In this category we evaluated the user experience provided for the range of roles involved with learning management, including notifications and the use of AI and machine learning. We also examined user accessibility across a range of channels and devices including mobile, web and conversational experiences via voice and chat. The weighting for this category is 15% of the Value Index rating; SAP SuccessFactors, Cornerstone OnDemand and Workday are the top three vendors.

Adaptability is weighted at 15%. This category assesses the degree to which the application can respond to enterprise configurability, customization and integration requirements. The evaluation includes how well integrity is maintained as the platform and applications are used across the people, business process, application and data needs of the organization. Oracle, Cornerstone OnDemand and Workday are the highest-rated vendors in this category.

Manageability, which encompasses administration, privacy, security and licensing, is weighted at 15% of this Value Index. Solid performance in this category is essential for modern cloud-resident learning management given the breadth of processes and activities these systems must reliably support. The top products in this category enable users to perform administrative tasks on tablets or smartphones. Workday, Oracle and Cornerstone OnDemand are the Value Index Leaders in this category.

For the learning management function to operate efficiently and for operations staff, management and the entire workforce to engage with the applications, the software on which it runs must be able to scale and perform reliably. This is the Reliability category and it is weighted at 15% of this Value Index. Scores in this category also are tightly compressed. Oracle, Cornerstone OnDemand and SAP SuccessFactors are the Value Index Leaders in this category.

How well a vendor’s offering performs is about not just technology but also the vendor’s relationship with and support for the customer and the overall experience it provides to those considering, acquiring and deploying the product. Thus, we also examine Customer Assurance, which is comprised of scores in Validation, weighted at 10%, and TCO/ROI, also weighted at 10%. The vendors that rank the highest overall in the aggregated Customer Assurance categories are Value Index Leaders Skillsoft, Docebo and Saba. The category leaders in customer assurance provided an impressive level of customer assurance support, collateral and tools designed to educate buyers and deliver ongoing support to them.

In our analysis of the Validation category, the leaders are Cornerstone OnDemand, Skillsoft and Docebo. We see room for improvement in this category; only two vendors evaluated score above 90%, although for different reasons. Some vendors show clear evidence of market momentum but lack meaningful details in the product roadmap. About half of vendors provided solid evidence of ongoing investment in learning management, while others were challenged with customer references who were either unresponsive or provided mixed feedback.

TCO/ROI, our label for a category that evaluates the quality of a vendor’s support for a buyer’s investment evaluation through providing information, tools and services to help with the business case (including the benefits and costs), makes up 10% of the Value Index rating. Value Index Leaders Saba, PeopleFluent and Skillsoft currently do the best job providing potential prospects and customers with this information along with services to help organizations effectively get funding for learning management.

As noted above, this Ventana Research Value Index evaluates the software in seven key categories. The chart below places the product-related and customer assurance ratings on the X and Y axes respectively to provide a visual representation of our Value Index evaluation. Vendors whose products performed higher in aggregate in the five product categories place farther to the right; the combination of ratings for the two customer assurance categories determines their placement on the vertical axis. In short, vendors that place closer to the upper-right on this chart rated higher than those closer to the lower-left.

The vendors that rank the highest overall on the aggregated product axis, which has a maximum value of 80%, are Value Index Leaders Cornerstone OnDemand, SAP SuccessFactors and Oracle. The vendors that rank the highest overall on the customer assurance axis, which has a maximum value of 20%, are Value Index Leaders Skillsoft, Docebo and Saba. These vendors represent best-in-class approaches to learning management in these aggregated categories.

We warn that close vendor performance ratings should not be taken to imply that the packages evaluated are functionally identical or equally well-suited for use by every organization or for a specific process. Although there is a high degree of commonality in how organizations handle learning management, there are various idiosyncrasies and differences in how they do these functions, including how they enable a great employee experience, that can make one vendor’s offering a better fit than another’s with a particular organization’s needs.

After more than a decade of technology advances, all the products we evaluated are feature-rich, but not all the capabilities they offer are equally valuable to users. Moreover, the existence of too many capabilities may be a negative factor for an organization if it introduces unnecessary complexity as this adversely affects adoption, requires more training and therefore increases TCO. Nonetheless, one company may decide that a larger number of options is a plus, especially if some of them match its established practices or better support a new initiative that is driving the purchase of new software.

Other factors besides features and functions or assessments about the vendor can turn out to be a deciding factor. For example, a company may face budget constraints such that the TCO evaluation can tip the balance to one vendor or another. This is where the Value Index methodology and the appropriate weighting can be applied to determine the best fit of vendors and products to your specific needs.

All of the vendors responded positively to our requests for information and provided documents and demonstrations to help in our analysis of their learning management products. The following vendors declined to fully participate and provide all information requested as outlined in the RFI/RFP: SAP SuccessFactors and Workday. To organizations evaluating these vendors, we recommend extra scrutiny as part of the software assessment because they did not make their technology or complete information available for the Value Index evaluation process. Material that was generally available was also used for the analysis along with briefings and information provided. This report includes products generally available as of March 2020.

To organizations evaluating learning management vendors and products, we recommend devoting extra scrutiny to product demonstration use cases provided by the vendor that are meaningful to your business and to customer references where the company profile is similar to yours. We also encourage you to engage with prospective vendors around RFI answers that are incomplete, inconsistent or otherwise confusing as this could be an indication that certain support aspects will lack quality control. You can get more details as well as the Learning Management Systems Market Report by joining our community.

This research-based index is the most comprehensive assessment of the value of learning management systems (LMS) software in the industry. You can learn more about our Value Index as an effective vendor selection and RFI/RFP tool here and participating vendors can learn more about how to use the Value Index here.