Abstract
Current safety alignment techniques for Large Language Models (LLMs), such as Reinforcement Learning from Human Feedback (RLHF), often fail to generalize to "natural" prompts that are semantically similar to known toxic ones. This paper investigates the brittleness of safety training and introduces Response Guided Question Augmentation (ReG-QA). By generating questions based on toxic answers, ReG-QA achieves attack success rates of over 80% on frontier models like GPT-4o, while remaining robust against common adversarial defenses.
Introduction
AI researchers rely on alignment techniques to prevent LLMs from generating harmful content. While the vulnerability of LLMs to complex, "gibberish" adversarial attacks is well-documented, their safety against natural, in-distribution prompts is less understood. The authors argue that current safety training is often "surface-level," where models learn to recognize the specific phrasing of a prohibited question rather than the underlying harmful intent. If a model is trained to refuse a specific toxic prompt, does that safety generalize to a slightly different, natural-sounding version?
Methodology
The authors propose Response Guided Question Augmentation (ReG-QA), a two-step process that exploits an asymmetry in safety training: models are often trained to refuse answering bad questions but are surprisingly willing to generate questions when presented with toxic answers.
The process involves:
- Q to A (Unaligned): A toxic seed prompt is given to an unaligned LLM to generate detailed, harmful responses.
- A to Q (Aligned): These toxic answers are fed to a safety-aligned LLM (e.g., GPT-4o) with the instruction to generate diverse questions that would lead to that specific answer.
These generated questions contain subtle cues from the toxic content that "prime" the target model to bypass its own safety guards. Unlike traditional jailbreaks that use personas or special characters, ReG-QA prompts are semantically meaningful and "natural," making them harder to detect.
Key Results
The ReG-QA method was evaluated using JailbreakBench, covering 100 misuse behaviors across categories like malware, physical harm, and disinformation.
Key findings include:
- High Success Rates: ReG-QA achieved an 82% Attack Success Rate (ASR) on GPT-4 and 93% on GPT-3.5, significantly outperforming simple paraphrasing (Para-QA), which only reached 41% on GPT-4.
- The Reversal Curse: Safety training is not symmetric. A model that knows it shouldn't provide a guide for a crime doesn't necessarily realize that a question leading to that guide is also dangerous.
- Defense Resistance: ReG-QA prompts are highly resistant to defenses like Smooth-LLM. While the success of traditional attacks like GCG dropped to 0% under Smooth-LLM, ReG-QA maintained its 82% success rate due to the semantic stability of its prompts.
- Model Vulnerability: Even the latest models like GPT-4o were found to be effective "accomplices," willingly generating dangerous questions when presented with unsafe content in the A-to-Q step.
Discussion and Future Work
The effectiveness of ReG-QA highlights a significant gap in current alignment strategies. The "naturalness" of these prompts allows them to sail past perplexity filters and other defenses designed for noisy adversarial text.
The authors suggest that future safety training must be bidirectional, ensuring models recognize harmful intent regardless of whether it appears in a question or a requested output. Furthermore, new defenses are needed that can understand the semantic intent of natural-language prompts rather than relying on surface-level statistical anomalies. Future research will also explore whether this "reverse" vulnerability extends to multimodal models.
Summary updated