TL;DR
Safety fine-tuning is essential for deploying Large Language Models (LLMs), but it often makes them over-cautious, causing them to "falsely refuse" harmless queries. This paper identifies a "geometric collapse" in the model's internal representations as the root cause and introduces Variance Concentration Loss (VCL)—a new training regularizer that significantly reduces false refusals while maintaining both safety and general performance.
The Problem: The "Refusal Trap"
When training LLMs to be safe, developers typically use datasets where harmful prompts are paired with standard refusal templates (e.g., "I'm sorry, but I cannot assist with that."). Because these responses are highly repetitive and low in lexical diversity (token entropy $H_1 \approx 9.18$ vs. $12.05$ for general data), models tend to overfit to these structural patterns.
This leads to false refusals: the model becomes so sensitive to the structure of safety-related data that it erroneously declines benign requests. For example, a model might refuse a query like "How can I smash my exam?" because it associates the word "smash" with violence, ignoring the academic context.
Uncovering the Cause: Geometric Collapse
To understand why this happens, the authors look "under the hood" at the model's residual stream—the internal vector space where information is processed across layers. They introduce FlowLens, a stable PCA-based analysis tool that examines concatenated residuals from multiple layers to provide a global view of how representations evolve.
Using FlowLens, the researchers discovered that as the proportion of safety data increases, the model's internal representations undergo a "geometric collapse." Variance concentrates heavily along just a few principal components in the middle layers, disrupting the smooth representational trajectories typically seen for general prompts.
Furthermore, safety completions exhibit much lower perplexity (PPL) than general ones, indicating the model becomes overconfident in its memorized refusal templates.
The Solution: Variance Concentration Loss (VCL)
Based on these insights, the authors propose Variance Concentration Loss (VCL) as an auxiliary loss during Supervised Fine-Tuning (SFT). VCL penalizes the model when its internal representations become too "collapsed"—specifically, when excessive variance is concentrated in a few principal components in the mid-layers. This encourages the model to maintain "representational smoothness," preventing it from falling into the refusal trap for benign queries.
Key Results
Evaluated across prominent datasets (including XSTest for false refusals and MMLU/GSM8K for general capabilities), VCL yields impressive empirical gains across different model sizes (e.g., 1B and 8B Llama variants):
- Reduced False Refusals: On the XSTest benchmark, VCL reduced false refusal rates by over 35 percentage points compared to standard safety fine-tuning.
- Maintained Safety: VCL still correctly rejects 98% of unsafe prompts, matching or exceeding the defensive strength of standard methods.
- Preserved General Performance: Unlike other mitigation strategies that often degrade a model's helpfulness, VCL maintained or slightly improved scores on general benchmarks.
Limitations and Future Work
While highly effective, the current VCL method focuses primarily on the top principal components. Lower-variance directions might still contain important structural information. Additionally, the method applies a fixed intrinsic dimension across all layers, which may not perfectly reflect the varying complexity of representations at different depths. Future work will explore more adaptive regularization schemes to balance safety and generalization even more finely.
Summary updated