TL;DR
This paper provides the first mechanistic explanation for why supervised fine-tuning (SFT) on reasoning tasks makes language models brittle, degrading their general performance. They discover SFT achieves reasoning by repurposing existing features and unnecessarily suppressing others, offering a path towards training methods that enhance reasoning without sacrificing general capabilities.
What Problem Does This Paper Address?
Large language models (LLMs) are increasingly being fine-tuned to improve their reasoning abilities, particularly with techniques like "chain-of-thought" prompting. However, this fine-tuning often leads to a phenomenon called "brittleness": while reasoning performance skyrockets, general capabilities like factual recall or common sense plummet. This trade-off is a significant hurdle for deploying LLMs in real-world applications. Existing solutions to mitigate this brittleness are often opaque, leaving researchers without a clear understanding of why this happens. This paper aims to bridge this gap by providing a mechanistic explanation for SFT-induced brittleness.
Methods
The researchers employ a multi-pronged approach to dissect the inner workings of SFT:
- Fisher Information with Sparse Autoencoders (SAEs): This technique is used to identify specific features within the LLM's activations that are most crucial for reasoning. By analyzing how features are activated during reasoning versus non-reasoning tasks, they pinpoint "reasoning features."
- Crosscoders: This method allows for a systematic comparison of feature spaces between a base model and a fine-tuned model. It helps understand how fine-tuning transforms existing features, categorizing them into "base-only," "shared," and "SFT-only" features based on their importance in each model.
- Gradient Blocking: This novel intervention technique allows researchers to selectively "freeze" certain subsets of features during the fine-tuning process. By observing the model's performance when specific feature groups are prevented from being modified, they can establish causal links between feature transformations and model capabilities.
Datasets
The primary dataset used for training and evaluation is a curated set of 47,000 mathematics problems, combining low-difficulty problems from DeepScaler and high-difficulty problems from SimpleRL. For evaluation, they utilize a comprehensive suite of benchmarks across four categories:
- Math Reasoning: MATH500, AIME, OlympiadBench.
- Other Reasoning: LiveCodeBench, GPQA-Diamond, ACPBench, HeadQA.
- General Reasoning & Commonsense QA: CommonsenseQA, LogiQA, OpenBookQA, PIQA, RACE, SciQ, SocialIQa.
- Non-Reasoning: IFEval, MC-TACO.
They also conducted validation experiments on the Llama-3.1-8B model family.
Key Insights and Ideas
- SFT's Dual Mechanism: The core finding is that SFT doesn't magically create new reasoning abilities. Instead, it operates through two main mechanisms:
- Repurposing Shared Features: SFT leverages and modifies features that are already present in the base model and are useful for both reasoning and general tasks.
- Suppressing Base-Only Features: SFT actively suppresses features that are primarily responsible for general capabilities, which is an unnecessary side effect for achieving reasoning.
- Reasoning Features are "Amplified" Not "Created": The identified reasoning features are not entirely new. They exist in the base model but are amplified and repurposed by SFT. This is evidenced by their concentration in the "shared" or slightly "SFT-only" regions of the crosscoder analysis.
- Separable Mechanisms: Gradient blocking experiments demonstrate that the repurposing of shared features is necessary for reasoning, while the suppression of base-only features is not. Blocking shared features completely eliminates reasoning, whereas blocking base-only features preserves reasoning performance.
- Robustness is Tied to Feature Preservation: Alternative fine-tuning methods like RLVR, which show better robustness, tend to preserve more shared features, highlighting the importance of not overly disrupting the base model's capabilities.
Key Results
- Brittleness Confirmed: Standard SFT on Qwen3-4B-Base improved Math Reasoning by 62% but degraded Non-Reasoning capabilities by 47%.
- Feature Steering Success: Using their Fisher Information method, feature steering achieved a 3.46% performance gain on base models, outperforming existing methods.
- Gradient Blocking Proof:
- Blocking Shared features completely eliminated Math Reasoning.
- Blocking Base-only features preserved Math Reasoning performance (42.6%), demonstrating its suppression is unnecessary.
- Limitations of Blocking: While blocking base-only features preserves reasoning, it doesn't fully restore non-reasoning capabilities, suggesting other factors contribute to brittleness.
Limitations and Future Work
The authors acknowledge that perfect control over feature modification is still elusive. Gradient leakage can occur, where models find ways to modify even "protected" features. They also note that brittleness arises from both the repurposing of shared features and the suppression of base-only features, meaning simply unblocking base-only features doesn't fully solve the problem.
Future work should focus on developing "surgical training methods" that selectively modify reasoning-relevant features while preserving general capabilities. This could involve more nuanced gradient blocking or other targeted intervention strategies.
Summary updated