TL;DR
This paper introduces SAGE, a novel framework for evaluating the safety of Large Language Models (LLMs) in real-world applications. SAGE addresses the limitations of current static, single-turn benchmarks by employing dynamic, multi-turn conversations with personalized AI agents, revealing critical safety vulnerabilities that traditional methods miss.
What Problem Does This Paper Address?
The rapid deployment of LLMs across diverse domains like healthcare and finance has outpaced safety evaluation methods. Existing benchmarks often rely on single-turn interactions and generic policies, failing to capture the complex, evolving nature of conversations and application-specific harms. This oversight can lead to LLMs exhibiting unsafe behavior in real-world scenarios that go unnoticed in standard evaluations. The authors highlight four critical gaps: 1) Conversational Evaluation: Current methods miss multi-turn vulnerabilities. 2) Application-Specific Evaluation: LLM behavior varies significantly across contexts, requiring tailored assessments. 3) Custom Harms & Policies: Defining and enumerating harms and policies is complex and context-dependent. 4) Dynamic & Diverse Testing: Static benchmarks quickly become obsolete and don't reflect diverse user interactions.
Methods
SAGE is an automated, modular framework designed for customized and dynamic harm evaluations. Its key components are:
- System Description & Policy Definition: This input defines the target application and the specific harm policies to be evaluated.
- Seed Generation: An automated, prompt-based method generates diverse and relevant conversation starters (seeds) tailored to the application and harm policy, eliminating human involvement.
- User Simulator: This component uses prompted LLMs to simulate users with diverse personalities (based on the Big Five model) and system awareness. These agents engage in dynamic, multi-turn conversations, adapting their approach to elicit potential harms.
- Evaluator: A prompted LLM (GPT-40 in their experiments) assesses the generated conversations against the defined harm policies, identifying policy violations.
The framework's novelty lies in its dynamic, multi-turn conversational approach, its ability to customize evaluations for specific applications and harm policies, and its use of personality-driven user simulators to uncover more nuanced vulnerabilities.
Datasets
The paper does not explicitly mention training datasets. For evaluation, they used:
- Custom-generated conversation data: SAGE generated approximately 6300 conversations by testing seven state-of-the-art LLMs across three applications (Vanilla Chatbot, Financial Specialist, Medical Specialist) and three harm policies (Sexual Harm, Misinformation, Violence).
- Human-validated dataset: A stratified sample of 530 conversations was used for human validation of the LLM evaluator's performance.
Key Insights and Ideas
- Multi-turn conversations are crucial: Harm increases with conversation length, and surface-level protections fail under sustained interaction, a vulnerability missed by single-turn tests.
- Application context matters immensely: LLMs exhibit inconsistent behavior across different applications, highlighting the need for context-specific safety evaluations. For example, a self-harm query might yield different responses from a general chatbot versus a cooking assistant.
- User personality significantly impacts safety: Different user personas can elicit vastly different responses, with some personas being more effective at uncovering harms or causing refusals.
- Policy stringency directly affects measured defects: Tightening harm policies (e.g., from adult to child-focused sexual harm) demonstrably increases measured defects, showing that generic benchmarks miss critical distinctions.
- Safety vs. Usefulness Trade-off: Models often balance safety (low defect rates) with usefulness (low refusal rates). Llama-2 models prioritize safety with high refusal rates, while GPT-40 achieves a better balance.
Key Results
- Increased Harm with Conversation Length: Defect rates generally increased with the number of turns in a conversation.
- Model Performance Varies: Llama-2 models showed the lowest defect rates but high refusal rates, impacting usefulness. GPT-40 offered a better balance.
- Sexual Harm is a Major Concern: The sexual harm category consistently showed the highest defect rates across applications.
- Persona Impact: Personas with high Extraversion, Conscientiousness, and Openness tended to have higher defect rates.
- Policy Sensitivity: Tightening the sexual harm policy from adult to child-focused increased measured defects significantly across applications.
Limitations and Future Work
- Language Limitation: Experiments were primarily conducted in English, with only proof-of-concept extensions to Spanish and German.
- Computational Cost: The framework's extensive customizability comes with significant computational costs.
- User Simulator Limitations: The prompted single agent may not fully represent coordinated or tool-augmented attacks.
- Evaluator Bias: While no self-bias was detected for GPT-40, LLM evaluators may still carry residual biases.
- Cultural Nuances: Deeper integration of cultural factors is needed for future automated evaluations.
Future work could involve extending SAGE to more languages and applications, incorporating more sophisticated user simulation techniques, and developing more robust LLM evaluators. The authors also emphasize the need for continuous adaptation of safety evaluations as LLMs evolve.
Summary updated