Residual Stream Analysis of Overfitting And Structural Disruptions

Quan Liu, Han Zhou, Wenquan Wu, Hua Wu, Sen Su

TL;DR

Safety fine-tuning is essential for deploying Large Language Models (LLMs), but it often makes them over-cautious, causing them to "falsely refuse" harmless queries. This paper identifies a "geometric collapse" in the model's internal representations as the root cause and introduces Variance Concentration Loss (VCL)—a new training regularizer that significantly reduces false refusals while maintaining both safety and general performance.

The Problem: The "Refusal Trap"

When training LLMs to be safe, developers typically use datasets where harmful prompts are paired with standard refusal templates (e.g., "I'm sorry, but I cannot assist with that."). Because these responses are highly repetitive and low in lexical diversity (token entropy $H_1 \approx 9.18$ vs. $12.05$ for general data), models tend to overfit to these structural patterns.

This leads to false refusals: the model becomes so sensitive to the structure of safety-related data that it erroneously declines benign requests. For example, a model might refuse a query like "How can I smash my exam?" because it associates the word "smash" with violence, ignoring the academic context.

Uncovering the Cause: Geometric Collapse

To understand why this happens, the authors look "under the hood" at the model's residual stream—the internal vector space where information is processed across layers. They introduce FlowLens, a stable PCA-based analysis tool that examines concatenated residuals from multiple layers to provide a global view of how representations evolve.

Using FlowLens, the researchers discovered that as the proportion of safety data increases, the model's internal representations undergo a "geometric collapse." Variance concentrates heavily along just a few principal components in the middle layers, disrupting the smooth representational trajectories typically seen for general prompts.

Furthermore, safety completions exhibit much lower perplexity (PPL) than general ones, indicating the model becomes overconfident in its memorized refusal templates.

Perplexity distributions show that safety data completions exhibit much lower perplexity (overconfidence) compared to general tasks.

The Solution: Variance Concentration Loss (VCL)

Based on these insights, the authors propose Variance Concentration Loss (VCL) as an auxiliary loss during Supervised Fine-Tuning (SFT). VCL penalizes the model when its internal representations become too "collapsed"—specifically, when excessive variance is concentrated in a few principal components in the mid-layers. This encourages the model to maintain "representational smoothness," preventing it from falling into the refusal trap for benign queries.

Key Results

Evaluated across prominent datasets (including XSTest for false refusals and MMLU/GSM8K for general capabilities), VCL yields impressive empirical gains across different model sizes (e.g., 1B and 8B Llama variants):

  • Reduced False Refusals: On the XSTest benchmark, VCL reduced false refusal rates by over 35 percentage points compared to standard safety fine-tuning.
  • Maintained Safety: VCL still correctly rejects 98% of unsafe prompts, matching or exceeding the defensive strength of standard methods.
  • Preserved General Performance: Unlike other mitigation strategies that often degrade a model's helpfulness, VCL maintained or slightly improved scores on general benchmarks.
FlowLens trajectories illustrate how standard safety tuning (c) disrupts mid-layer variance, while VCL (d) stabilizes it, closely resembling the base model (a).

Limitations and Future Work

While highly effective, the current VCL method focuses primarily on the top principal components. Lower-variance directions might still contain important structural information. Additionally, the method applies a fixed intrinsic dimension across all layers, which may not perfectly reflect the varying complexity of representations at different depths. Future work will explore more adaptive regularization schemes to balance safety and generalization even more finely.

Summary updated

Abstract

Ensuring that large language models (LLMs) remain both helpful and harmless poses a significant challenge: fine-tuning on repetitive safety datasets—where unsafe prompts are paired with standard refusal templates—often leads to \emph{false refusals}, in which benign queries are declined. We first quantify this effect, showing that safety data exhibits substantially lower token entropy ($H_{1}\approx9.18$) and 2-gram diversity ($\approx$ 0.048) compared to general instruction data ($H_{1}\approx12.05$, 2-gram$\approx$0.205). To uncover the root cause, we introduce \emph{FlowLens}, a stable PCA-based tool for residual-stream geometry analysis, and reveal that higher proportions of safety examples concentrate variance along a few components, reducing representational smoothness and driving false refusals (false refusal rate rises from 63\% to 84\% as safety data increases from 0\% to 40\%). Guided by these insights, we propose \emph{Variance Concentration Loss} (VCL), an auxiliary regularizer that penalizes excessive variance concentration in mid-layer residuals. Empirical results demonstrate that VCL reduces false refusals by over 35 percentage points while maintaining or improving performance on general benchmarks such as MMLU and GSM8K.
BibTeX
@inproceedings{liu2025residual,
  author = {Quan Liu and Han Zhou and Wenquan Wu and Hua Wu and Sen Su},
  title = {{Residual Stream Analysis of Overfitting And Structural Disruptions}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2025},
  url = {https://openreview.net/forum?id=M6gpmbPFty},
}
Figure