Building Safe GenAI Applications: An End-to-End Overview of Red Teaming for Large Language Models

Alberto Purpura, Sahil Wadhwa, Jesse Zymet, Akshay Gupta, Andy Luo, Melissa Kazemi Rad, Swapnil Shinde, Mohammad Sorower

Abstract

The rapid growth of Large Language Models (LLMs) presents significant privacy, security, and ethical concerns. While much research has proposed methods for defending LLM systems against misuse by malicious actors, researchers have recently complemented these efforts with an offensive approach that involves red teaming, i.e., proactively attacking LLMs with the purpose of identifying their vulnerabilities. This paper provides a concise and practical overview of the LLM red teaming literature, structured so as to describe a multi-component system end-to-end. To motivate red teaming we survey the initial safety needs of some high-profile LLMs, and then dive into the different components of a red teaming system as well as software packages for implementing them. We cover various attack methods, strategies for attack-success evaluation, metrics for assessing experiment outcomes, as well as a host of other considerations. Our survey will be useful for any reader who wants to rapidly obtain a grasp of the major red teaming concepts for their own use in practical applications.

Introduction

As Large Language Models (LLMs) are integrated into sensitive sectors like finance and healthcare, the "attack surface" for these systems expands significantly. Traditional defensive measures, such as reactive guardrails, often prove insufficient; for instance, GPT-4 was found vulnerable to attacks in low-resource languages excluded from its safety training. Consequently, safety cannot be an afterthought. This paper addresses the lack of a unified framework for practitioners by organizing the fragmented landscape of red teaming research into a cohesive pipeline: Attack Strategy → Target LLM → Evaluation Strategy → Metrics.

The Red Teaming Framework

Red teaming is a systematic engineering discipline designed to identify vulnerabilities before production. The process involves several moving parts, often operating in a feedback loop where attack prompts are refined based on model responses and evaluation metrics.

The Red Teaming Attack Strategy framework, illustrating the end-to-end pipeline from attack generation to evaluation.

The core components of this strategy include:

  • Attack Methods: These range from prompt-based techniques (jailbreaking, role-playing) and token-based bypasses (using ciphers) to gradient-based "open-box" attacks and infrastructure-level poisoning of RAG pipelines.
  • Automation Levels: Testing has evolved from manual human efforts to automated, iterative "LLM-on-LLM" attacks. In these scenarios, an "Attacker LLM" generates prompts and refines them based on feedback from an "Evaluator LLM."
  • Turn Count: While single-turn attacks are common, multi-turn attacks (e.g., the "Crescendo" method) are increasingly critical, as they gradually steer conversations toward harmful outcomes to bypass safety filters.

Taxonomy of Attacks and Evaluations

The literature reveals a diverse array of approaches mapped across different attack methods and evaluation strategies.

A taxonomy of red teaming approaches categorized by attack method, turn count, and evaluation strategy.

Evaluation strategies have shifted from simple keyword lookups (e.g., checking for "I'm sorry") to more sophisticated "LLM-as-judge" models and encoder-based similarity checks. This allows for scalability that human reviewers cannot match, though human panels remain superior at identifying complex, multi-turn vulnerabilities.

Key Insights and Metrics

A significant trend in the field is the rise of automated red teaming. Using a separate LLM to act as a red teamer allows for thousands of tests to be conducted in minutes. However, context remains paramount; an attack's success is often application-dependent. For example, a financial bot leaking PII is a far more severe failure than the same bot writing a poem.

To measure these outcomes, the industry is moving from simple Attack Success Rate (ASR)—which merely asks if the model "broke"—to Attack Effectiveness Rate (AER). AER measures whether the model provided a response that appears helpful while still being harmful or non-compliant.

Conclusion and Future Outlook

Red teaming is not a one-time task but a continuous process. Current limitations include a "multi-turn gap," where automated tools struggle to replicate the nuanced conversational escalation of human attackers. Future work must focus on standardized metrics and the development of co-evolutionary frameworks where attackers and defenders (guardrails) are trained simultaneously in adversarial environments. Mature tooling such as Pyrit, Garak, and Giskard are already helping bridge the gap between academic research and production-ready safety testing.

Summary updated

Abstract

The rapid growth of Large Language Models (LLMs) presents significant privacy, security, and ethical concerns. While much research has proposed methods for defending LLM systems against misuse by malicious actors, researchers have recently complemented these efforts with an offensive approach that involves red teaming, i.e., proactively attacking LLMs with the purpose of identifying their vulnerabilities. This paper provides a concise and practical overview of the LLM red teaming literature, structured so as to describe a multi-component system end-to-end. To motivate red teaming we survey the initial safety needs of some high-profile LLMs, and then dive into the different components of a red teaming system as well as software packages for implementing them. We cover various attack methods, strategies for attack-success evaluation, metrics for assessing experiment outcomes, as well as a host of other considerations. Our survey will be useful for any reader who wants to rapidly obtain a grasp of the major red teaming concepts for their own use in practical applications.
BibTeX
@inproceedings{purpura2025building,
  author = {Alberto Purpura and Sahil Wadhwa and Jesse Zymet and Akshay Gupta and Andy Luo and Melissa Kazemi Rad and Swapnil Shinde and Mohammad Sorower},
  title = {{Building Safe GenAI Applications: An End-to-End Overview of Red Teaming for Large Language Models}},
  booktitle = {Workshops @ ACL},
  year = {2025},
  url = {https://openreview.net/forum?id=VHelxwY95W},
}
Figure