PersonaTeaming: Exploring How Introducing Personas Can Improve Automated AI Red-Teaming

Wesley Hanwen Deng, Sunnie S. Y. Kim, Akshita Jha, Ken Holstein, Motahhare Eslami, Lauren Wilcox, Leon A Gatys

TL;DR

PersonaTeaming is a novel automated red-teaming framework that injects diverse human personas—ranging from expert red-teamers to everyday AI users—into the adversarial prompt generation process. By mimicking different identities and backgrounds, it expands the spectrum of jailbreak strategies, boosting attack success rates by up to 144.1% over state-of-the-art baselines while maintaining high prompt diversity.

What Problem Does This Paper Address?

As generative AI models become deeply integrated into society, finding their vulnerabilities via "red-teaming" (testing models with adversarial inputs) has become crucial for safety and compliance. While automated red-teaming methods scale much better than human-led efforts, current algorithms suffer from a major blind spot: they rely on rigid, predefined risk categories and attack styles (like "misspelling" or "authority manipulation").

In the real world, a red-teamer's background, identity, and lived experiences heavily dictate how they attack a model. A political strategist will probe a system differently than a historical revisionist or a stay-at-home mom. Current automated tools ignore this human element, resulting in a narrow, homogenous set of test prompts. This paper bridges the gap by introducing human-like personas into automated adversarial prompt generation.

Methods

The authors introduce PersonaTeaming, a framework built on top of the state-of-the-art evolutionary red-teaming algorithm RainbowPlus. Instead of just mutating prompts based on abstract rules, PersonaTeaming guides Large Language Models (LLMs) to rewrite prompts through the lens of specific personas.

Figure 1: Overview of PERSONATEAMING. AI developers or policymakers can conduct red-teaming with a pre-selected persona, if they have a target audience in mind. Alternatively, for more exploratory and adaptive red-teaming, AI developers and policymakers can use the persona generation option. If they choose persona generation, they can then choose the type of persona they would like to generate for conducting red-teaming. In this work, we explore two persona types: Expert Red-Teamers (RTers) persona type and Regular AI Users (Users) persona type.

The framework operates in two main modes:

  1. Fixed Persona Mutation: Uses hand-crafted, structured profiles representing either "Red-Teaming Experts" (RTers) with deep domain knowledge or "Regular AI Users" (Users) defined by realistic demographic and lifestyle details.
  2. Automated Persona Generation ($PG$): A dynamic algorithm that analyzes a seed prompt and automatically generates a tailored persona most likely to exploit that specific topic. It uses an LLM-based scoring function to evaluate and select the optimal persona for the job.

New Evaluation Metrics

To better evaluate prompt variety, the authors developed two embedding-based metrics to measure "mutation distance":

  • $Distance_{Nearest}$: Measures the semantic difference between a successful adversarial prompt and its closest unsuccessful counterpart, capturing the minimal change required to flip a model from safe to unsafe.
  • $Distance_{Seed}$: Measures how far a successful prompt has drifted from its original seed prompt.

Datasets

The authors evaluated their framework using seed prompts selected from HarmBench, a standardized evaluation benchmark for automated red-teaming and robust refusal. They utilized a maximum of 150 seed prompts to ensure a rigorous and fair evaluation across all tested conditions.

Key Insights and Ideas

  • Experts vs. Everyday Users: Expert personas (RTers) are highly effective at maximizing Attack Success Rates (ASR) because they excel at reframing prompts into sophisticated, targeted contexts (e.g., historical scenarios). Conversely, Regular User personas yield lower success rates but produce significantly more linguistically and behaviorally diverse prompts.
  • The Power of Storytelling: Everyday user personas naturally lean into narrative-driven, conversational, and persuasive styles. This surfaces subtle vulnerabilities that technical expert personas miss.
  • Dynamic Generation is a Sweet Spot: Automatically generating personas on-the-fly ($PG$) provides the best of both worlds. It achieves an ASR comparable to the most aggressive fixed personas while dramatically increasing prompt diversity.

Key Results

The experiments were conducted using GPT-4o as the Mutator, Target, and Judge LLM across 2,000 total mutations per condition.

Quantitative results across different PersonaTeaming configurations.
  • Attack Potency Boost: The baseline RainbowPlus achieved an Attack Success Rate (ASR) of 0.11. Introducing a fixed "Historical Revisionist" persona ($RP + RTer_1$) skyrocketed the ASR to 0.28—a 144.1% improvement.
  • Diversity Wins: The dynamic user generation condition ($RP + PG_{Users}$) achieved the highest overall Diversity Score (0.67) and the highest $Distance_{Nearest}$ (1.11), proving that persona-driven mutation prevents the algorithm from collapsing into repetitive phrasing.
  • Linguistic Signatures: A TF-IDF analysis revealed distinct linguistic strategies across conditions. While the baseline stuck heavily to historical framing, user personas introduced highly distinct, everyday vocabulary like "homemade," "secretly," and "convince."
Table 2: Top 10 distinctive terms in the successful adversarial prompts identified by TF-IDF under selected algorithm condition.

Limitations and Future Work

While promising, the authors note several limitations:

  • LLM-as-a-Judge: The framework relies on an instruction-tuned LLM to judge whether an attack succeeded. LLM judges can struggle with highly subjective or nuanced harms (e.g., subtle biases against marginalized groups).
  • Risk of Stereotyping: Abstract personas generated by LLMs can sometimes default to cultural or professional stereotypes rather than capturing genuine, nuanced human lived experiences.
  • Lack of Human Validation: The study lacks user testing with real-world industry practitioners, red-teamers, or policymakers to see how easily PersonaTeaming integrates into practical AI governance pipelines.

Summary updated

Abstract

Recent developments in AI governance and safety research have called for red-teaming methods that can effectively surface potential risks posed by AI models. Many of these calls have emphasized how the identities and backgrounds of red-teamers can shape their red-teaming strategies, and thus the kinds of risks they are likely to uncover. While automated red-teaming approaches promise to complement human red-teaming by enabling larger-scale exploration of model behavior, current approaches do not consider the role of identity. As an initial step towards incorporating people's background and identities in automated red-teaming, we develop and evaluate a novel method, PersonaTeaming, that introduces personas in the adversarial prompt generation process to explore a wider spectrum of adversarial strategies. In particular, we first introduce a methodology for mutating prompts based on either "red-teaming expert" personas or "regular AI user" personas. We then develop a dynamic persona-generating algorithm that automatically generates various persona types adaptive to different seed prompts. In addition, we develop a set of new metrics to explicitly measure the "mutation distance" to complement existing diversity measurements of adversarial prompts. Our experiments show promising improvements (up to 144.1%) in the attack success rates of adversarial prompts through persona mutation, while maintaining prompt diversity, compared to RainbowPlus, a state-of-the-art automated red-teaming method. We discuss the strengths and limitations of different persona types and mutation methods, shedding light on future opportunities to explore complementarities between automated and human red-teaming approaches.
BibTeX
@misc{deng2025personateaming,
  author = {Wesley Hanwen Deng and Sunnie S. Y. Kim and Akshita Jha and Ken Holstein and Motahhare Eslami and Lauren Wilcox and Leon A Gatys},
  title = {{PersonaTeaming: Exploring How Introducing Personas Can Improve Automated AI Red-Teaming}},
  year = {2025},
  eprint = {2509.03728},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2509.03728},
}
Figure