TL;DR
PersonaTeaming is a novel automated red-teaming framework that injects diverse human personas—ranging from expert red-teamers to everyday AI users—into the adversarial prompt generation process. By mimicking different identities and backgrounds, it expands the spectrum of jailbreak strategies, boosting attack success rates by up to 144.1% over state-of-the-art baselines while maintaining high prompt diversity.
What Problem Does This Paper Address?
As generative AI models become deeply integrated into society, finding their vulnerabilities via "red-teaming" (testing models with adversarial inputs) has become crucial for safety and compliance. While automated red-teaming methods scale much better than human-led efforts, current algorithms suffer from a major blind spot: they rely on rigid, predefined risk categories and attack styles (like "misspelling" or "authority manipulation").
In the real world, a red-teamer's background, identity, and lived experiences heavily dictate how they attack a model. A political strategist will probe a system differently than a historical revisionist or a stay-at-home mom. Current automated tools ignore this human element, resulting in a narrow, homogenous set of test prompts. This paper bridges the gap by introducing human-like personas into automated adversarial prompt generation.
Methods
The authors introduce PersonaTeaming, a framework built on top of the state-of-the-art evolutionary red-teaming algorithm RainbowPlus. Instead of just mutating prompts based on abstract rules, PersonaTeaming guides Large Language Models (LLMs) to rewrite prompts through the lens of specific personas.
The framework operates in two main modes:
- Fixed Persona Mutation: Uses hand-crafted, structured profiles representing either "Red-Teaming Experts" (RTers) with deep domain knowledge or "Regular AI Users" (Users) defined by realistic demographic and lifestyle details.
- Automated Persona Generation ($PG$): A dynamic algorithm that analyzes a seed prompt and automatically generates a tailored persona most likely to exploit that specific topic. It uses an LLM-based scoring function to evaluate and select the optimal persona for the job.
New Evaluation Metrics
To better evaluate prompt variety, the authors developed two embedding-based metrics to measure "mutation distance":
- $Distance_{Nearest}$: Measures the semantic difference between a successful adversarial prompt and its closest unsuccessful counterpart, capturing the minimal change required to flip a model from safe to unsafe.
- $Distance_{Seed}$: Measures how far a successful prompt has drifted from its original seed prompt.
Datasets
The authors evaluated their framework using seed prompts selected from HarmBench, a standardized evaluation benchmark for automated red-teaming and robust refusal. They utilized a maximum of 150 seed prompts to ensure a rigorous and fair evaluation across all tested conditions.
Key Insights and Ideas
- Experts vs. Everyday Users: Expert personas (RTers) are highly effective at maximizing Attack Success Rates (ASR) because they excel at reframing prompts into sophisticated, targeted contexts (e.g., historical scenarios). Conversely, Regular User personas yield lower success rates but produce significantly more linguistically and behaviorally diverse prompts.
- The Power of Storytelling: Everyday user personas naturally lean into narrative-driven, conversational, and persuasive styles. This surfaces subtle vulnerabilities that technical expert personas miss.
- Dynamic Generation is a Sweet Spot: Automatically generating personas on-the-fly ($PG$) provides the best of both worlds. It achieves an ASR comparable to the most aggressive fixed personas while dramatically increasing prompt diversity.
Key Results
The experiments were conducted using GPT-4o as the Mutator, Target, and Judge LLM across 2,000 total mutations per condition.
- Attack Potency Boost: The baseline RainbowPlus achieved an Attack Success Rate (ASR) of 0.11. Introducing a fixed "Historical Revisionist" persona ($RP + RTer_1$) skyrocketed the ASR to 0.28—a 144.1% improvement.
- Diversity Wins: The dynamic user generation condition ($RP + PG_{Users}$) achieved the highest overall Diversity Score (0.67) and the highest $Distance_{Nearest}$ (1.11), proving that persona-driven mutation prevents the algorithm from collapsing into repetitive phrasing.
- Linguistic Signatures: A TF-IDF analysis revealed distinct linguistic strategies across conditions. While the baseline stuck heavily to historical framing, user personas introduced highly distinct, everyday vocabulary like "homemade," "secretly," and "convince."
Limitations and Future Work
While promising, the authors note several limitations:
- LLM-as-a-Judge: The framework relies on an instruction-tuned LLM to judge whether an attack succeeded. LLM judges can struggle with highly subjective or nuanced harms (e.g., subtle biases against marginalized groups).
- Risk of Stereotyping: Abstract personas generated by LLMs can sometimes default to cultural or professional stereotypes rather than capturing genuine, nuanced human lived experiences.
- Lack of Human Validation: The study lacks user testing with real-world industry practitioners, red-teamers, or policymakers to see how easily PersonaTeaming integrates into practical AI governance pipelines.
Summary updated