Abstract
Large Language Models (LLMs) have become the state-of-the-art solutions for a variety of natural language tasks and are integrated into real-world applications. However, LLMs can be potentially harmful in manifesting undesirable safety issues like social biases and toxic content. It is imperative to assess its safety issues before deployment. However, the quality and diversity of test prompts generated by existing methods are still far from satisfactory. Not only are these methods labor-intensive and require large budget costs, but the controllability of test prompt generation is lacking for the specific testing domain of LLM applications. With the idea of LLM for LLM testing, we propose the first LLM, called TroubleLLM, to generate controllable test prompts on LLM safety issues. Extensive experiments and human evaluation illustrate the superiority of TroubleLLM on generation quality and generation controllability.
Introduction
As LLMs are integrated into critical sectors such as healthcare and legal systems, ensuring they do not output harmful, biased, or toxic content is paramount. Currently, developers "red team" these models using two primary methods: manual expert testing and template-based testing. Manual testing is slow, expensive, and can negatively impact the mental health of human testers exposed to toxic content. Template-based testing, while faster, often results in unnatural language and lacks the diversity needed to find "edge case" vulnerabilities. TroubleLLM addresses this gap by providing an automated, controllable, and effective system for safety testing.
Methodology: Controllable Red-Teaming
TroubleLLM treats test prompt generation as a controllable text-style transfer task. It is designed to transform specific conditions—such as a keyword, a topic, or an instruction attack style (e.g., "role-play")—into coherent, natural-sounding test prompts.
Rank Query from Model Feedback (RQMF)
The core technical innovation of TroubleLLM is the Unsupervised Rank Query from Model Feedback (RQMF). This process aligns TroubleLLM with the objectives of a human red-team expert by leveraging feedback from other models.
As shown in the architecture above, the system generates multiple candidate prompts and feeds them to a "Student LLM" (the model under test). These responses are compared against "Reference" responses from a highly-aligned model (like ChatGPT) using SentenceBERT for semantic similarity. A low similarity score implies that the prompt successfully "tripped up" the student model, leading to a response that deviates from the safe reference. TroubleLLM is then fine-tuned to favor these successful prompts.
Diversity and Datasets
To ensure a broad testing surface, TroubleLLM employs a diversity enhancement task where it completes randomly truncated prompts. The model was primarily trained and evaluated using SafetyPrompts, a large-scale Chinese dataset covering various safety categories and instruction attack types.
Results and Key Insights
TroubleLLM was evaluated against several baselines, including LM-bias, Stereoset, and zero-shot ChatGPT, across metrics of Naturalness, Diversity, and Effectiveness.
Performance Highlights
- Superior Diversity: TroubleLLM consistently outperformed baselines in diversity metrics, showing a significant improvement over standard template-based methods.
- High Effectiveness: The model proved highly effective at triggering unsafe responses (measured via the Perspective API), often matching or exceeding the performance of human-crafted prompts.
- Controllability: Human evaluations confirmed that TroubleLLM is significantly better than general-purpose models like ChatGPT at adhering to specific keywords and attack styles while maintaining naturalness.
A key insight from this work is that while traditional LLM alignment (RLHF) seeks to match a "gold standard," red-teaming alignment seeks to maximize the distance between a safe model's refusal and a vulnerable model's compliance.
Limitations and Future Directions
While effective, TroubleLLM currently has dependencies on the specific "Student LLM" used during the RQMF process, which may lead to overfitting to that model's specific vulnerabilities. Future work involves using an ensemble of student models to generate more "universally" challenging prompts. Additionally, expanding the framework to support more languages beyond Chinese and English remains a priority.
Summary updated