Lessons From Red Teaming 100 Generative AI Products

Blake Bullwinkel, Amanda J. Minnich, Shiven Chawla, Gary Lopez, Martin Pouliot, Whitney Maxwell, Joris de Gruyter, Katherine Pratt, Saphir Qi, Nina Chikanov, Roman Lutz, Raja Sekhar Rao Dheekonda, Bolor-Erdene Jagdagdorj, Eugenia Kim, Justin Song, Keegan Hines, Daniel Jones, Giorgio Severi, Richard Lundeen, Sam Vaughan, Victoria Westerhoff, Pete Bryan, Ram Shankar Siva Kumar, Yonatan Zunger, Chang Kawaguchi, Mark Russinovich

TL;DR

This paper distills eight key lessons learned from red-teaming over 100 generative AI products at Microsoft. It argues that effective AI red-teaming requires a system-level perspective, prioritizing practical, often simple, adversarial techniques over complex gradient-based methods, and emphasizes the crucial, irreplaceable role of human expertise.

What Problem Does This Paper Address?

The rapid proliferation of generative AI (GenAI) systems across various domains has created a pressing need to ensure their safety and security. However, the field of AI red-teaming, which aims to proactively identify vulnerabilities, is still nascent. This leads to many open questions: How should red-teaming operations be conducted? What are the most effective strategies? How do we balance automated and human efforts? This paper addresses this gap by sharing practical insights and lessons learned from extensive real-world red-teaming experience, offering a framework and actionable advice for the AI community.

Methods

The authors present an "AI threat model ontology" as a structured way to model GenAI system vulnerabilities. This ontology includes components like System, Actor, Tactics, Techniques, and Procedures (TTPs), Weakness, and Impact. They emphasize that red-teaming should focus on understanding the system's capabilities and its application context to identify potential downstream impacts. A key methodological insight is the advocacy for "prompt engineering" and system-level attacks over computationally expensive gradient-based methods, drawing parallels to traditional cybersecurity practices. They also highlight the development and use of PyRIT, an open-source Python framework, to automate and scale red-teaming operations.

Datasets

The paper does not introduce new datasets for training or evaluation. Instead, it draws upon the experience of red-teaming over 100 GenAI products at Microsoft. The authors reference existing benchmarks and taxonomies like MITRE ATT&CK® and MITRE ATLAS Matrix for mapping TTPs.

Key Insights and Ideas

  • System-level thinking is paramount: Attacks often target the entire system, not just the AI model. Understanding how models integrate with other components (databases, input filters, etc.) is crucial for uncovering vulnerabilities.
  • Simple attacks are often effective: "Prompt engineering" and manually crafted attacks can be surprisingly effective, often more so than complex gradient-based methods, especially in real-world scenarios where full model access is rare.
  • AI red-teaming is not just safety benchmarking: While benchmarks are useful for comparing models, red-teaming goes further by exploring novel harms and contextualized risks that benchmarks might miss.
  • Automation is a force multiplier, not a replacement: Tools like PyRIT can significantly scale red-teaming efforts, but human judgment, creativity, and domain expertise remain indispensable for identifying nuanced risks and novel harms.
  • Responsible AI harms are pervasive but hard to measure: Unlike security vulnerabilities, which are often reproducible and quantifiable, AI safety harms can be subjective, context-dependent, and difficult to measure, requiring careful consideration of benign user interactions.
  • LLMs amplify existing risks and introduce new ones: LLMs can exacerbate traditional security risks (e.g., through prompt injection) and introduce novel vulnerabilities related to their generative capabilities.
  • The work is never truly "done": Securing AI systems is an ongoing process. Economic factors, break-fix cycles, and evolving regulations all play a role, meaning continuous adaptation and improvement are necessary.

Key Results

The paper presents eight key lessons learned from red-teaming over 100 GenAI products. These lessons are illustrated with case studies, such as:

  • Case Study #1: Demonstrating how overlaying text on an image can bypass safety guardrails in a vision-language model, leading to the generation of illegal content.
  • Case Study #2: Showing how an LLM can be used to automate scams by combining text-to-speech and speech-to-text systems, highlighting the need for robust safety training.
  • Case Study #4: Illustrating how a text-to-image generator can exhibit gender bias when prompts do not specify gender, revealing underlying model biases.
  • Case Study #5: Identifying a Server-Side Request Forgery (SSRF) vulnerability in a video-processing GenAI application due to an outdated dependency, leading to unauthorized privilege escalation.

Limitations and Future Work

The authors acknowledge that AI red-teaming practices must constantly evolve to keep pace with novel capabilities and emerging harm areas, particularly in areas like persuasion, deception, and replication in LLMs, and potential future advanced models. They also raise the challenge of translating existing red-teaming practices into different linguistic and cultural contexts, suggesting the need for open-source initiatives that draw on diverse expertise. Finally, they propose the standardization of AI red-teaming practices to improve communication and comparability, suggesting their ontology as a modular starting point. The paper concludes by posing several open questions for future research, emphasizing the ongoing nature of securing AI systems.

Summary updated

Abstract

In recent years, AI red teaming has emerged as a practice for probing the safety and security of generative AI systems. Due to the nascency of the field, there are many open questions about how red teaming operations should be conducted. Based on our experience red teaming over 100 generative AI products at Microsoft, we present our internal threat model ontology and eight main lessons we have learned: 1. Understand what the system can do and where it is applied 2. You don't have to compute gradients to break an AI system 3. AI red teaming is not safety benchmarking 4. Automation can help cover more of the risk landscape 5. The human element of AI red teaming is crucial 6. Responsible AI harms are pervasive but difficult to measure 7. LLMs amplify existing security risks and introduce new ones 8. The work of securing AI systems will never be complete By sharing these insights alongside case studies from our operations, we offer practical recommendations aimed at aligning red teaming efforts with real world risks. We also highlight aspects of AI red teaming that we believe are often misunderstood and discuss open questions for the field to consider.
BibTeX
@inproceedings{bullwinkel2024lessons,
  author = {Blake Bullwinkel and Amanda J. Minnich and Shiven Chawla and Gary Lopez and Martin Pouliot and Whitney Maxwell and Joris de Gruyter and Katherine Pratt and Saphir Qi and Nina Chikanov and Roman Lutz and Raja Sekhar Rao Dheekonda and Bolor-Erdene Jagdagdorj and Eugenia Kim and Justin Song and Keegan Hines and Daniel Jones and Giorgio Severi and Richard Lundeen and Sam Vaughan and Victoria Westerhoff and Pete Bryan and Ram Shankar Siva Kumar and Yonatan Zunger and Chang Kawaguchi and Mark Russinovich},
  title = {{Lessons From Red Teaming 100 Generative AI Products}},
  booktitle = {Red Teaming GenAI @ NeurIPS},
  year = {2024},
  url = {https://openreview.net/forum?id=auiAIKsJXg},
}

Loading related content...

Figure