Why Does Reinforcement Learning Generalize? A Feature-Level Mechanistic Study of Post-Training in Large Language Models

Dan Shi, Zhuowen Han, Simon Ostermann, Renren Jin, Josef van Genabith, Deyi XiongarXiv 2026 · arXiv

TL;DR

While supervised fine-tuning (SFT) often causes large language models to forget general capabilities, reinforcement learning (RL) helps them generalize far beyond their training domain. This paper uses a novel "Three-Model Sparse Crosscoder" to show that SFT aggressively forces models to learn rigid, specialized features, whereas RL makes restrained, continuous adjustments that preserve the base model's core representations. Crucially, the authors discover and causally validate a compact, task-agnostic set of internal features that directly govern this cross-task generalization.

What Problem Does This Paper Address?

When post-training large language models (LLMs), researchers face a striking paradox:

  1. Supervised Fine-Tuning (SFT), which trains models on full reasoning trajectories, often leads to "catastrophic forgetting" or degradation of general capabilities.
  2. Reinforcement Learning (RL), which typically relies only on sparse, outcome-level rewards (e.g., right or wrong answers), somehow enables models to generalize robustly to entirely unseen, out-of-domain tasks.

Why does a weaker, indirect signal (RL) yield broader, more transferable improvements than explicit supervision (SFT)? To answer this, the authors move past behavioral observations and dive into a feature-level mechanistic analysis of the model's internal activation spaces.

Methods

To ensure a fair comparison, the authors set up a controlled experiment: they trained an SFT model and an RL model from the exact same base model using identical data.

To analyze how features evolve, they introduced two primary interpretability tools:

  • Two-Model and Three-Model Sparse Crosscoders: Standard Sparse Autoencoders (SAEs) find interpretable features in a single model. Crosscoders extend this by jointly encoding activations from multiple sources into a shared feature space. The authors developed a Three-Model Sparse Crosscoder to align the activation spaces of the Base, SFT, and RL models simultaneously.
  • Model Attribution Score (MAS): A novel metric designed to quantify feature specificity across the three models, allowing researchers to see exactly which training paradigm a feature belongs to.

To find the root cause of generalization, they isolated generalization-critical samples—prompts where the base model failed but the RL model succeeded. By analyzing feature activations on these samples across diverse tasks, they identified a compact intersection of task-agnostic features. They then performed causal interventions: zeroing out these features in the RL model and amplifying them in the base model to verify their functional roles.

Datasets

  • Training: A high-quality mathematics dataset consisting of 47K problems derived from MATH and DeepScaler. For SFT, the targets were Chain-of-Thought (CoT) traces from Qwen3-32B-Instruct. For RL, the model was optimized via Group Relative Policy Optimization (GRPO) using outcome-level correctness rewards.
  • Evaluation:
  • In-domain Math Tasks: MATH500, AIME24, AIME25.
  • Out-of-domain General Tasks: OpenBookQA, CommonsenseQA, HeadQA, SciQ, ARC-Challenge.
  • Unseen Transfer Tasks: LogiQA, PIQA (used to test the robustness of the discovered generalization features).

Key Insights and Ideas

  • SFT is a Rigid Overhaul; RL is a Gentle Reweighting: SFT rapidly introduces a massive number of highly specialized features early in training while suppressing the base model's original capabilities. RL, conversely, introduces very few exclusive features. It works primarily by subtly reweighting and preserving the base model's existing representational structure.
Figure 1: Distribution of Normalized Relative Norms across different training methods and different model scales.
  • Continuous Exploration vs. Early Stabilization: Feature rank-shift analysis revealed that SFT features stabilize almost immediately. RL models, however, exhibit continuous feature turnover throughout training, meaning the RL process constantly explores and reconfigures which internal features are most salient for generating correct outcomes.
  • Generalization is Driven by a Task-Agnostic Circuit: The features that help the RL model solve a math problem are largely the same features that help it solve a commonsense science question. Generalization isn't about learning task-specific heuristics; it's driven by a compact, universal set of internal control features.
  • The Knowledge is Already There: Amplifying generalization features in the base model unlocked massive performance gains on tasks it previously failed. This proves that post-training doesn't necessarily teach the model new facts; rather, RL acts as a control signal that effectively activates latent upstream circuitry already present in the base model.

Key Results

  • Behavioral Discrepancies: On CommonsenseQA, the Qwen3-4B-RL model outperformed its SFT counterpart by 31.0%, despite being trained purely on math problems.
  • Causal Ablation (Zeroing Out): When the authors deactivated the small set of identified generalization features in the RL-tuned models, performance plummeted. For instance, the Qwen3-4B-RL model suffered a 46.2% drop on OpenBookQA and a 43.9% drop on CommonsenseQA.
  • Causal Amplification: Forcing these generalization features to activate in the unmodified base models led to massive performance spikes. On SciQ, amplifying these features allowed Qwen2.5-7B to correctly answer 55.6% of the samples it had previously failed.

Limitations and Future Work

While this study provides a compelling mechanistic look under the hood of post-training, the authors note a couple of limitations:

  1. Crosscoder Completeness: Relying on Sparse Crosscoders means the analysis is bound to the learned sparse feature space, which is not guaranteed to capture all functionally relevant internal structures.
  2. Actionable Training Objectives: Although inference-time feature interventions successfully steer model behavior, these insights do not yet directly translate into a formula for designing new training objectives or loss functions that explicitly encourage generalization features during training. The authors leave this bridge from interpretation to design for future work.

Summary updated

Abstract

Reinforcement learning (RL)-based post-training often improves the reasoning performance of large language models (LLMs) beyond the training domain, while supervised fine-tuning (SFT) frequently leads to general capabilities forgetting. However, the mechanisms underlying this contrast remain unclear. To bridge this gap, we present a feature-level mechanistic analysis methodology to probe RL generalization using a controlled experimental setup, where RL- and SFT-tuned models are trained from the same base model on identical data. Leveraging our interpretability framework, we align internal activations across models within a shared feature space and analyze how features evolve during post-training. We find that SFT rapidly introduces many highly specialized features that stabilize early in training, whereas RL induces more restrained and continually evolving feature changes that largely preserve base models' representations. Focusing on samples where RL succeeds but the base model fails, we identify a compact, task-agnostic set of features that directly mediate generalization across diverse tasks. Feature-level interventions confirm their causal role: disabling these features significantly degrades RL models' generalization performance, while amplifying them improves base models' performance. The code is available at https://github.com/danshi777/RL-generalization.
BibTeX
@misc{shi2026why,
  author = {Dan Shi and Zhuowen Han and Simon Ostermann and Renren Jin and Josef van Genabith and Deyi Xiong},
  title = {{Why Does Reinforcement Learning Generalize? A Feature-Level Mechanistic Study of Post-Training in Large Language Models}},
  year = {2026},
  eprint = {2604.25011},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2604.25011},
}

Loading related content...

Figure