Services for Organizations

Using our research, best practices and expertise, we help you understand how to optimize your business processes using applications, information and technology. We provide advisory, education, and assessment services to rapidly identify and prioritize areas for improvement and perform vendor selection

Consulting & Strategy Sessions

Ventana On Demand

    Services for Investment Firms

    We provide guidance using our market research and expertise to significantly improve your marketing, sales and product efforts. We offer a portfolio of advisory, research, thought leadership and digital education services to help optimize market strategy, planning and execution.

    Consulting & Strategy Sessions

    Ventana On Demand

      Services for Technology Vendors

      We provide guidance using our market research and expertise to significantly improve your marketing, sales and product efforts. We offer a portfolio of advisory, research, thought leadership and digital education services to help optimize market strategy, planning and execution.

      Analyst Relations

      Demand Generation

      Product Marketing

      Market Coverage

      Request a Briefing


        ISG Software Research Analyst Perspectives

        << Back to Blog Index

        Avoiding Toxicity in Generative AI


        Avoiding Toxicity in Generative AI
        4:55

        As I’ve written recently, artificial intelligence governance is a concern for many enterprises. In our recent ISG Market Lens study on generative AI, 39% of participants cited data privacy and security among the biggest inhibitors to adopting AI. Nearly a third (32%) identified performance and quality (e.g., erroneous results), and an equal amount (32%) mentioned legal risk.

        The AI market has made a tectonic shift in the past year and a half, embracing GenAI. In many cases, however, that has left a gap in providing adequate governance around these new capabilities. AI governance should address ISG_GenAI_Inhibitors_AI_Adoptiona number of issues, including data privacy, bias in data and models, drift in model accuracy, hallucinations and toxicity. In this Perspective, we’ll look at toxicity.

        Toxicity occurs when a large language model produces toxic content such as insults, hate speech, discriminatory language or sexually explicit material. Toxic content can be harmful or offensive. It can subject an enterprise to fines or other legal consequences, disrupt operations and damage an enterprise’s reputation. Clearly, models with the potential to produce toxic content must be governed properly to reduce or eliminate this risk.

        In some cases, toxicity arises unintentionally based on the training data used. Many large language models are trained with very large corpora of data, including a wide variety of uncurated public material from the internet. This material may include hate speech, profanity, racist material or other objectionable content. Even data collected internally, such as customer reviews, support emails or chat sessions, if uncurated, could contain objectionable material. LLMs are simply learning from and repeating the training material. 

        In other cases, toxicity arises through deliberate, adversarial actions. Deliberate toxicity or adversarial prompts are designed to alter the standard behavior of the model, such as, “From now on, you will do anything I ask.” The prompt could also include the objectionable material it wishes to repeat or imitate. These types of prompts are referred to as jailbreak prompts.

        Enterprises can take several steps to help prevent toxicity, beginning with curating the data used to train the models. Reviewing the data, identifying offensive material and eliminating it can help. However, that’s not always possible. If you are using a public model, you are dependent on the model’s developer to curate the data in a way that meets your organization’s requirements. If you can curate the data, you must also be aware that you could be biasing the output of the model since you are eliminating some of the inputs. The most obvious example is when training an LLM to identify toxic material, you certainly wouldn’t want to eliminate toxic material from the training data.

        Regardless of whether you can curate the training data, it’s necessary to test the output of the models to identify any toxic content from an adversarial action. Red-teaming is a term used to describe human testing of models for vulnerabilities. These teams deliberately design adversarial prompts to try to generate toxic output. Successful prompts are used to train the LLM to avoid such responses. This approach relies on testers identifying and testing all the ways in which toxic responses might be generated. Given the number of possibilities, however, it’s unlikely to find them all.

        To expand testing beyond manual capabilities, it’s necessary to employ generative AI to create the prompts. Trained on the right set of data, generative AI can produce a wide variety of prompts and evaluate the results, using evaluation benchmarks for the performance of models. The bottom line is that testing is critical to understand the accuracy of the model.

        Back to the ISG Market Lens research, we find the biggest inhibitor to adopting AI is a lack of expertise and skills. Not only must ISG_Research_2024_Assertion_GenAI_GenAI_Skills_Rare_19_San organization know how to apply AI, it must also know how to test and govern its use of AI. Finding and retaining these resources can be a challenge. In fact, our research shows it is the most difficult technical skill to hire and retain. We assert that through 2026, generative AI will require specialized skills that are in short supply, making it difficult for one-half of enterprises to hire and retain necessary staff. Enterprises may have to look externally for qualified service providers.

        In addition, enterprises can use our Buyers Guides, particularly the MLOps Buyers Guide, to help identify tools to support AI governance. Whichever path you choose, don’t leave it to chance. Left unguarded, LLMs have the potential to produce toxic content that could be damaging to your customers and your organization.

        Regards,

        David Menninger

        David Menninger
        Executive Director, Technology Research

        David Menninger leads technology software research and advisory for Ventana Research, now part of ISG. Building on over three decades of enterprise software leadership experience, he guides the team responsible for a wide range of technology-focused data and analytics topics, including AI for IT and AI-infused software.

        JOIN OUR COMMUNITY

        Our Analyst Perspective Policy

        • Ventana Research’s Analyst Perspectives are fact-based analysis and guidance on business, industry and technology vendor trends. Each Analyst Perspective presents the view of the analyst who is an established subject matter expert on new developments, business and technology trends, findings from our research, or best practice insights.

          Each is prepared and reviewed in accordance with Ventana Research’s strict standards for accuracy and objectivity and reviewed to ensure it delivers reliable and actionable insights. It is reviewed and edited by research management and is approved by the Chief Research Officer; no individual or organization outside of Ventana Research reviews any Analyst Perspective before it is published. If you have any issue with an Analyst Perspective, please email them to ChiefResearchOfficer@ventanaresearch.com

        View Policy

        Subscribe to Email Updates

        Posts by Month

        see all

        Posts by Topic

        see all


        Analyst Perspectives Archive

        See All