Using our research, best practices and expertise, we help you understand how to optimize your business processes using applications, information and technology. We provide advisory, education, and assessment services to rapidly identify and prioritize areas for improvement and perform vendor selection
We provide guidance using our market research and expertise to significantly improve your marketing, sales and product efforts. We offer a portfolio of advisory, research, thought leadership and digital education services to help optimize market strategy, planning and execution.
Services for Technology Vendors
We provide guidance using our market research and expertise to significantly improve your marketing, sales and product efforts. We offer a portfolio of advisory, research, thought leadership and digital education services to help optimize market strategy, planning and execution.
Embracing artificial intelligence technologies opens doors for innovation and efficiency. Alongside these opportunities, however, come risks. Threat actors are keenly aware of the potential impact of AI systems and are actively exploring ways to manipulate them. In this Analyst Perspective, I explore the world of adversarial machine-learning threats and provide practical guidance for securing AI systems.
Generative AI applications introduce great potential to enterprises, bringing the intersection of innovation and security closer to business needs. Ventana Research asserts that by 2026, one-third of enterprises will establish governance to mitigate risks associated with GenAI software, ensuring ethical considerations, avoiding data and model bias, and safeguarding privacy and data security.
GenAI systems can inadvertently generate biased or harmful content. Enterprises must ensure that ethical guidelines are in place to direct AI-generated outputs and prevent unintended consequences. GenAI models also learn from historical data, which may contain biases. Organizations need robust mechanisms to detect and rectify bias during model training and deployment. Plus, GenAI applications often process sensitive data. Implementing strong privacy controls and safeguarding data against breaches is essential.
Striking a balance between innovation and digital security becomes paramount for enterprise adoption of GenAI software. Let’s explore the adversarial threats that enterprises must navigate to protect AI systems.
A recent report by scientists from the National Institute of Standards and Technology and its collaborators identifies vulnerabilities in AI systems that adversaries can use to create malfunctions. The NIST report provides an overview of four major attack types that AI systems might suffer and corresponding approaches to mitigate damage. The report further classifies attacks according to multiple criteria such as the attacker's goals and objectives, capabilities and knowledge. The leading attack types include abuse, evasion, poisoning and privacy.
Abuse attacks involve exploiting AI systems for unintended purposes. Using a language model to generate harmful or inappropriate content by compromising legitimate data sources is an example. Strict access controls, content filtering and regular audits can prevent abuse attacks.
Evasion attacks aim to deceive AI models during inference by subtly modifying input data. For example, adversaries alter an image, such as a speed limit sign or road lane marker, to mislead an image-recognition system into misclassifying objects. Robust model architectures, input preprocessing and adversarial training can help mitigate evasion attacks.
Poisoning attacks involve injecting malicious data into the training set to compromise model performance. An attacker, for example, modifies training data to bias a sentiment analysis model like a chatbot into utilizing inappropriate language. Rigorous data validation, outlier detection and monitoring during training are essential to prevent poisoning attacks.
Privacy attacks exploit model outputs to infer sensitive data about individuals. A membership inference attack reveals whether a specific data point was part of the training set. By asking a chatbot legitimate questions, an attacker may use those responses to identify and exploit a weakness in the model. Differential privacy techniques, model aggregation and data anonymization enhance privacy protection.
While enterprises invest significant effort in building robust AI models, attackers face lower barriers. Building attacks on AI systems requires less effort than constructing a robust AI model. Attackers can exploit vulnerabilities without the complexity of model development, training and optimization. Enterprises must recognize this asymmetry and prioritize AI security measures.
A wealth of open-source tools and research papers are available for crafting attacks. Adversarial attack libraries, pre-trained models and tutorials are accessible online. Organizations must defend against these documented techniques.
Threat actors can test their attacks against various models without rigorous validation. Enterprises often lack comprehensive testing for adversarial scenarios. Rigorous testing, including adversarial examples, is essential during model development.
The potential impact of compromising AI systems drives attacker’s motivation. Enterprises may underestimate the attractiveness of AI assets to attackers. Organizations should assess the value of AI models from an adversary’s perspective.
As organizations embrace AI, understanding the unique security challenges becomes paramount. Understanding the lower barriers faced by attackers is crucial for effective AI security. Enterprises must proactively address these challenges to safeguard AI systems and data. Essential practices to safeguard AI assets include:
The convergence of AI’s transformative potential and the increasing sophistication of threat actors requires a proactive approach to safeguarding AI systems. Securing AI systems is an ongoing journey that demands collaboration and adaptability. By following these best practices, enterprises can secure the AI watering hole and contribute to a safer digital future.
Regards,
Jeff Orr
Jeff Orr leads the research and advisory for the CIO and digital technology expertise at ISG Software Research, with a focus on modernization and transformation for IT. Jeff’s coverage spans cloud computing, DevOps and platforms, digital security, intelligent automation, ITOps and service management, intelligent automation and observation technologies across the enterprise.
Ventana Research’s Analyst Perspectives are fact-based analysis and guidance on business,
Each is prepared and reviewed in accordance with Ventana Research’s strict standards for accuracy and objectivity and reviewed to ensure it delivers reliable and actionable insights. It is reviewed and edited by research management and is approved by the Chief Research Officer; no individual or organization outside of Ventana Research reviews any Analyst Perspective before it is published. If you have any issue with an Analyst Perspective, please email them to ChiefResearchOfficer@ventanaresearch.com