Services for Organizations

Using our research, best practices and expertise, we help you understand how to optimize your business processes using applications, information and technology. We provide advisory, education, and assessment services to rapidly identify and prioritize areas for improvement and perform vendor selection

Consulting & Strategy Sessions

Ventana On Demand

    Services for Investment Firms

    We provide guidance using our market research and expertise to significantly improve your marketing, sales and product efforts. We offer a portfolio of advisory, research, thought leadership and digital education services to help optimize market strategy, planning and execution.

    Consulting & Strategy Sessions

    Ventana On Demand

      Services for Technology Vendors

      We provide guidance using our market research and expertise to significantly improve your marketing, sales and product efforts. We offer a portfolio of advisory, research, thought leadership and digital education services to help optimize market strategy, planning and execution.

      Analyst Relations

      Demand Generation

      Product Marketing

      Market Coverage

      Request a Briefing

        Ventana Research Analyst Perspectives

        << Back to Blog Index

        The Diminishing Science of Research in Technology Analyst Firms

        The technology analyst industry has ended its barrage of exuberant predictions for the new year. After digesting all of these commentaries (some were a challenge, I admit), we need to stop our predicting and instead start a discussion about what all of us ought to be doing. From my perspective, that should be an annual statement of direction, what we here call the research agenda. I have decided it is time to remind all of us what this important activity is and why it is really important to see from industry analysts.

        It should be a standard practice in our business for each firm to issue a research agenda that is supposed to guide its analysts or practices in the coming year. Ours have been available to clients and now will become public record as they are annually. Unfortunately, most analyst firms have excused themselves from performing this important step; instead, they just bring forward predictions and what I have heard from many of my colleagues as educated guesses from their opinion and inquiries from dozens of clients that are looking to advance their technology. The absence of a research agenda is not a good idea, for the simplest of reasons: Clients pay for services, and they have a right to expect that the analysts whose time they pay for will set out what their knowledge and experience leads them to conclude are the areas on which they should focus so the clients can evaluate how it matches their own objectives.

        At Ventana Research we are guided in setting our research agenda by what our clients need to advance their competency and maturity. In practice, for each market segment this requires comprehensive research across hundreds of organizations. We do this using our benchmarking methodology, which provides a framework for conducting, analyzing and presenting the research. Using this approach we are able to evaluate the level of maturity of an organization in a particular realm and also to drill into the maturity of its specific people, process, information and technology competencies. Another required element of how we establish competency and maturity is that we conduct research in both the business and IT sides of an organization. Researching only one of these two audiences can significantly skew the research findings and thus guidance provided to clients and the industry.

        We’re hardly the only firm that claims to conduct research. But we are unique in how our research is conducted within a consistent framework and is guided by clearly stated overall objectives. I’ve pointed out years ago (See: “Why Bad Research Could Jeopardize Your Business”) that conducting surveys without a methodology and not as part of an overall research agenda can easily rob the results of actionable value. Our benchmark research, in contrast, provides us with a strong foundation on which to build our education efforts and to use in advising business and IT buyers.

        Our research and the insights it yields makes it possible for us to determine best and worst practices, providing a perspective on what to do and what to avoid. Most analyst firms fall far short in identifying these because they do not conduct primary research, depending instead only on client inquiries and in most cases exclusively from IT. Unfortunately, this can bias their view of the market. Recently one of the largest IT analyst firms, one that does not do primary research across business and IT, admitted publicly that it was late to the table in covering mobile technologies. It did not see that the business side is driving the adoption of mobile technologies in the enterprise because it only advises the IT side, which has not yet been pushing to adopt these in more than a corporate standard for electronic mail.

        Nor is just doing buyer research sufficient. One needs to understand the technology itself in order to evaluate its fit for the people and process needs of an organization. Doing so requires not just supplier/vendor briefings and attending product demonstrations and reading product reviews but actually engaging into trying the combination of software and technology. It still amazes me that the majority of analysis who do not like to actually see, try and evaluate the technology they offer advice. It is important that analyst firms offering advice and guidance have undertaken a detailed assessment of different vendors and their products to assess their merits in order to be able to match them to buyer needs and evaluate how well they address the overall market.

        We do this using an approach that is framed as a balanced RFP (request for proposal). It includes evaluation categories such as usage, capabilities and management as well as cost, benefit and an examination of other customers’ deployments. Some approaches spend more time looking at the vision of the vendor and the future of their products or looking at their financials and number of customers. While these are important, they should not comprise much more than 20 percent of your evaluation review of vendors and products. If you are weighting these vendor criteria much more than this amount, you could be doing your business and technology environment a disservice.

        Why I am starting a conversation on the science of research and what you should expect from technology analyst firms? The answer is to let you, the buyer, know that there is a wide spectrum of research methods across analyst firms, and to ensure that you understand this to be able to determine the value of the services and advice you are receiving. Does the analyst with which you work have sufficient experience in the segment or technology that matters to you? Does that analyst derive knowledge from in-depth buyer research, supplier research, or both? Does his or her perspective come from being a counselor to organizations or only from reading research and education that is already published? Does the advice you’re given speak directly to your current competencies and those of your organization, or does it just regurgitate advice drawn from a spread of technology adoption laggards to leaders?

        All of these are questions you should ask yourself to ensure you get the advice you need – and, more importantly, the quality of knowledge transfer that can enable you to decide your needs and to act on them quickly. It is time for you to be more knowledgeable on what you are hearing and reading to ensure you can trust the information and advice you are utilizing for your business and IT organizations. I may be biased in my framing of this dialogue, of course, but after 20 years of seeing the good, the bad and the ugly of analyst firms and worked for one other than Ventana Research, I thought it would be good to start a discussion on what you as a buyer should expect in analyst firms and their research to provide you with the service you need and deserve.

        Let me know your thoughts or come and collaborate with me on Facebook, LinkedIn and Twitter.


        Mark Smith – CEO & EVP Research


        Mark Smith
        Partner, Head of Software Research

        Mark Smith is the Partner, Head of Software Research at ISG and Ventana Research leading the global market agenda as a subject matter expert in digital business and enterprise software. Mark is a digital technology enthusiast using market research and insights to educate and inspire enterprises, software and service providers.


        Our Analyst Perspective Policy

        • Ventana Research’s Analyst Perspectives are fact-based analysis and guidance on business, industry and technology vendor trends. Each Analyst Perspective presents the view of the analyst who is an established subject matter expert on new developments, business and technology trends, findings from our research, or best practice insights.

          Each is prepared and reviewed in accordance with Ventana Research’s strict standards for accuracy and objectivity and reviewed to ensure it delivers reliable and actionable insights. It is reviewed and edited by research management and is approved by the Chief Research Officer; no individual or organization outside of Ventana Research reviews any Analyst Perspective before it is published. If you have any issue with an Analyst Perspective, please email them to

        View Policy

        Subscribe to Email Updates

        Posts by Month

        see all

        Posts by Topic

        see all

        Analyst Perspectives Archive

        See All