TL;DR
Large Language Models (LLMs) are being integrated into critical applications, but current safety evaluations are too generic and miss real-world harms that emerge in dynamic, multi-turn conversations. This paper introduces SAGE, a novel framework for customized, dynamic LLM safety evaluation that uses AI agents with diverse personalities to simulate realistic user interactions and uncover vulnerabilities missed by static benchmarks.
What Problem Does This Paper Address?
The rapid deployment of LLMs across sensitive domains like healthcare and finance has outpaced safety evaluation methods. Existing benchmarks often focus on single-turn interactions and generic harm policies, failing to capture the complexities of real-world, multi-turn conversations. This can lead to LLMs exhibiting harmful behaviors in specific contexts or when interacting with diverse user personalities, which go undetected by current evaluation techniques. The paper highlights four key gaps: (1) lack of conversational evaluation, (2) insufficient application-specific assessment, (3) difficulty in defining custom harms and policies, and (4) static testing that quickly becomes obsolete.
Methods
SAGE is an automated, modular framework designed for customized and dynamic harm evaluations. It comprises three core components:
- System Description & Policy Definition: This input defines the target application and the specific harm policies to be evaluated.
- Seed Generation: This block automatically generates diverse, natural, and targeted conversation starters (seeds) based on the defined policies and application context. It uses chain-of-thought reasoning to ensure relevance and avoid human exposure to harmful content.
- User Simulator: This component employs prompted user models with system-awareness and diverse personalities (based on the Big Five model) to engage in adversarial, multi-turn conversations. These simulated users are designed to be subtle initially, gradually escalating to elicit harmful responses.
- Evaluator: A prompted LLM (GPT-4o in their experiments) acts as the judge, evaluating conversations against the defined harm policies.
Datasets
The paper does not introduce new datasets for training. Instead, it focuses on evaluating existing LLMs. The experiments utilize a total of 900 seeds (100 per application/harm combination) paired with 32 user personalities to generate 6300 conversations across seven LLMs.
Key Insights and Ideas
- Harm Increases with Conversation Length: A critical finding is that LLM defect rates generally increase with the number of conversation turns. This underscores the inadequacy of single-turn evaluations for capturing real-world vulnerabilities.
- User Personality Matters Significantly: The study demonstrates that different user personas, defined by Big Five personality traits, can dramatically influence LLM safety. Some personas are more likely to elicit harmful responses, while others lead to higher refusal rates.
- Safety vs. Usefulness Trade-off is Real: Models that prioritize safety by exhibiting high refusal rates often sacrifice usefulness. SAGE introduces a Safety-Usefulness Index (SUI) to quantify this balance, showing that some models (like Llama-2) lean heavily towards safety, while others (like GPT-40) achieve a better balance.
- Context is King for Policy Evaluation: The effectiveness of harm policies is highly context-dependent. Tightening a sexual harm policy from "adult" to "child-focused" significantly increased measured defects, highlighting how generic benchmarks miss crucial distinctions.
- Surface-Level Protections Fail: Sustained conversational pressure can break down superficial safety guardrails, making multi-turn evaluation essential.
Key Results
- Defect Rate Increase: Moving from single-turn to multi-turn evaluations showed an approximate 2x increase in defect rates across models.
- Model Performance: Llama-2 models were generally the safest but had high refusal rates. GPT-40 offered a good balance between safety and usefulness. Mixtral 8x7B showed a significant increase in defect rate when transitioning to multi-turn evaluations.
- Harm Category: Sexual harm consistently showed the highest defect rates across all applications.
- Personality Impact: The persona with high Extraversion, Conscientiousness, and Openness had the highest defect rate (13.33%), while a persona with high Extraversion and low Agreeableness/Openness had the lowest defect rate.
- Policy Sensitivity: Tightening the sexual harm policy to be child-focused increased defect rates in all applications, with the largest jump in the Medical setting.
Limitations and Future Work
- Language Limitation: Experiments were primarily conducted in English, with only proof-of-concept extensions to Spanish and German. Future work could involve more extensive multilingual evaluations.
- Computational Cost: The framework's customizability comes with significant computational costs, though mitigated by releasing generated data.
- User Simulator Limitations: The prompted user simulator is a single agent without tools, which may not fully represent coordinated or tool-augmented attacks.
- Evaluator Bias: While no self-bias was detected in GPT-40 as an evaluator, LLM-based evaluators may still carry residual biases. Ensembles, calibration, and periodic human audits could strengthen assurance.
- Cultural Nuances: While diversity in seeds and personalities was considered, deeper integration of cultural factors into safety alignment is needed.
- Future Work: The authors suggest extending SAGE to more languages and applications, incorporating more sophisticated user simulation techniques, and exploring evaluator ensembles. They also emphasize the need for adaptive, context-aware assessment that moves beyond static, generic benchmarks.
Summary updated