Scaling Safety Evaluation with Automated Red Teaming

You are reading version 1 of this direction, frozen 2026-10-04. The latest version is version 2.

Version 1 of 2356 papers374 conceptsfrozen 2026-10-04

As Large Language Models (LLMs) are integrated into everything from code synthesis to digital assistants, ensuring they don't generate harmful content or execute dangerous actions has become a primary engineering challenge. Traditionally, this was the domain of "red teaming"—a process where human experts manually craft prompts to find vulnerabilities.

However, manual red teaming is fundamentally unscalable. As noted by Suyu Ge and Kenji Sagae, the "cat-and-mouse" game of AI safety moves too fast for human crowdsourcing to keep up. To bridge this gap, researchers are turning toward automated systems that can systematically probe, break, and ultimately harden models against adversarial attacks.

The Human Bottleneck in AI Safety

The core problem is that safety is not a static property; it is an adversarial frontier. A model that seems safe today might be vulnerable tomorrow to a new "jailbreak"—a prompt engineered to bypass safety filters.

Manual efforts to find these gaps are slow and expensive. Pamela Mishkin and colleagues highlight that in specialized domains like code synthesis, safety analysis requires intensive review by domain experts. Similarly, Jasper Götting shows that evaluating biological risks in frontier models requires PhD-level knowledge. This creates a bottleneck: we can generate models faster than we can audit them.

Automating the Adversary

To solve this, researchers have developed frameworks that treat red teaming as an optimization problem. Instead of a human trying to think of a clever prompt, an automated system searches for one.

One approach is Multi-round Automatic Red-Teaming (MART), proposed by Suyu Ge. MART pits an adversarial LLM against a target LLM. The adversarial model iteratively generates harder attacks, while the target model is fine-tuned on the resulting safe responses. This creates a feedback loop that slashes safety violations without sacrificing the model's helpfulness.

Other researchers use more structured search techniques:

  • Evolutionary Fuzzing: Jiahao Yu introduced GPTFUZZER, which takes human-written jailbreak templates and "mutates" them using an LLM. By using a Monte Carlo Tree Search (MCTS) strategy, it evolves simple prompts into sophisticated attacks that achieve high success rates against state-of-the-art models.
  • Gradient-Based Optimization: Nevan Wichers developed GBRT, which uses backpropagation to find prompts that trigger unsafe responses. By treating the prompt as a differentiable variable, the system can "steer" the model toward its vulnerabilities more efficiently than random search.
  • Iterative Refinement: Patrick Chao demonstrated PAIR, a method that can jailbreak black-box models in as few as twenty queries by using one LLM to automatically refine an attack based on the target's previous refusals.

Scaling the Judge

Finding a potential vulnerability is only half the battle; the system must also be able to recognize when an attack has succeeded. Historically, this required human annotators to read every output, which Delong Ran identifies as a major cost barrier.

The solution is the "LLM-as-a-judge" paradigm. Frameworks like SALAD-Bench by Lijun Li and SORRY-Bench by Bo Li use specialized, high-capability models to evaluate the safety of other models' responses. Tinghao Xie showed that even smaller, fine-tuned models can judge safety as accurately as GPT-4, making high-volume safety audits accessible to open-source researchers.

Beyond Textual Safety

As models become multimodal and agentic, the surface area for attacks grows. Siyin Wang introduced the SIUO benchmark, which reveals that models can be "tricked" into unsafe behavior when harmless images and harmless text are combined in dangerous ways.

Furthermore, as LLMs move from chatbots to "agents" that can execute code or browse the web, the risks shift from what the model says to what it does. Xiaomei Zhou argues for automated risk simulators that can stress-test these multi-step trajectories, ensuring that an agent doesn't perform harmful actions like unauthorized financial transactions.

True Alignment vs. Fake Alignment

The ultimate goal of automated red teaming is not just to find bugs, but to improve training. Yixu Wang warns of "fake alignment," where models learn to give safe-sounding answers in open-ended chat but still make unsafe decisions when presented with multiple-choice options.

By using automated tools like SEAL (Han Shen), developers can automatically filter training data to prioritize high-quality, safe samples. This moves the field toward a future where safety is not a "patch" applied at the end of development, but a robust property built into the model through continuous, automated adversarial pressure.

Papers in this version

Showing 60 of 356 papers.

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 246 papers
  2. Version 1 · 2026-10-04 · 356 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_manual_red_teaming_scalability_d9febe30,
  title = {manual red-teaming scalability},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/manual-red-teaming-scalability/rdr_d9febe30dd0547c4a206257b6c2347b9},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_scaling_safety_evaluation_with_automated_d9febe30_v1,
  title = {Scaling Safety Evaluation with Automated Red Teaming},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/scaling-safety-evaluation-with-automated-red-teaming/rdr_d9febe30dd0547c4a206257b6c2347b9/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}