Who Reasons in the Large Language Models?

Jie Shao, Jianxin Wu

Abstract

While Large Language Models (LLMs) demonstrate remarkable reasoning capabilities, the internal localization of these functions remains poorly understood. This paper introduces Stethoscope for Networks (SfN), a diagnostic framework designed to probe the internal mechanics of reasoning. Our findings reveal that the output projection module (o_proj) within the Transformer's attention mechanism acts as the primary engine for logical reasoning. We demonstrate that surgically swapping or fine-tuning only the o_proj module can transfer reasoning capabilities between models with high efficiency, suggesting a fundamental "Division of Labor" between reasoning and conversational modules.

Introduction

The emergence of reasoning in LLMs is often treated as a holistic property of the network. However, identifying specific "circuits" responsible for logic versus those responsible for linguistic fluency is critical for developing more efficient and interpretable models. If reasoning is localized, we can optimize training by targeting specific modules, thereby reducing computational costs and improving model steerability.

Methodology: Stethoscope for Networks (SfN)

To investigate the localization of reasoning, we developed the Stethoscope for Networks (SfN) framework, consisting of four diagnostic approaches:

  1. The Delta Stethoscope: Analyzes weight differences between base models and their reasoning-enhanced counterparts to identify modules undergoing the most significant structural shifts.
  2. The Merge Stethoscope: Tests the "Frankenstein" transferability of capabilities by surgically replacing specific modules in a base model with those from a reasoning model.
  3. The Freeze Stethoscope: Evaluates if specific modules can learn reasoning in isolation by freezing the rest of the network during fine-tuning.
  4. The Destruction Stethoscope: Systematically disables modules to observe the resulting degradation in logic versus conversational ability.
The Stethoscope for Networks (SfN) framework and the Division of Labor conjecture.

Results

The o_proj Singularity

Analysis across the Qwen2.5 and Llama-3 families reveals that the o_proj module consistently exhibits unique behavior. While most modules show unimodal weight changes during reasoning-oriented fine-tuning, o_proj displays a distinct bimodal distribution, indicating a fundamental reorganization of its weights.

Heatmap of weight changes across layers, highlighting the unique role of the o_proj module.

Surgical Transfer of Reasoning

Using the Merge Stethoscope, we found that replacing only the o_proj in a Qwen-1.5B base model with the version from DeepSeek-R1-Distill increased AIME 2024 accuracy from 6.7% to 20.0%. In contrast, replacing MLP modules led to a total collapse in performance, suggesting that o_proj is uniquely specialized for logic.

Efficiency and Performance

Fine-tuning only the o_proj and LayerNorm parameters (approximately 10% of the total) achieved performance parity with full-parameter fine-tuning on the Qwen2.5-32B model. This targeted approach was 3x faster and significantly more memory-efficient, while also providing greater training stability.

Discussion: The Division of Labor Conjecture

We propose a "Division of Labor" conjecture: reasoning is localized within the o_proj module, while conversational fluency and context maintenance are handled by the MLP and Q/K/V projection modules. Destruction tests support this, showing that models with re-initialized o_proj remain "chatty" but lose all logical depth, whereas destroying MLP modules results in incoherent output.

This localization suggests a "Plugin Hypothesis," where specialized o_proj modules could be swapped into base models to provide domain-specific expertise (e.g., math, law, medicine) without retraining the entire network.

Conclusion

Our work identifies the o_proj module as the localized seat of reasoning in dense Transformer LLMs. By leveraging the SfN framework, we demonstrate that reasoning can be treated as a modular "plugin," offering a path toward more efficient and interpretable AI development. Future research will explore whether these findings extend to Mixture-of-Experts (MoE) architectures.

Summary updated

BibTeX
@inproceedings{shao2025who,
  author = {Jie Shao and Jianxin Wu},
  title = {{Who Reasons in the Large Language Models?}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2025},
  url = {https://openreview.net/forum?id=XIqlxqNDCL},
}
Figure