TL;DR
This paper introduces a weight-based taxonomy to analyze how gated neurons in Large Language Models (LLMs) read from and write to the residual stream. It discovers a small but highly influential class of "weakening neurons" in late layers that actively suppress specific informational directions, exerting an outsize impact on model behavior even during negative gating states.
What Problem Does This Paper Address?
Mechanistic interpretability aims to reverse-engineer the inner workings of transformer models. While previous research has categorized neurons based on when they activate or what tokens they promote, it often overlooks the direct relationship between what a neuron reads from the residual stream and what it writes back.
This input-output (IO) relationship is especially complex in modern LLMs (like Llama, Gemma, and OLMo) that use Gated Linear Units (GLUs) such as SwiGLU or GEGLU. Because GLU neurons involve three weight vectors (input, gate, and output), understanding their joint functionality requires a more sophisticated framework than traditional ReLU networks. This paper addresses this gap by establishing a formal IO taxonomy for gated neurons to understand how they manipulate representations across different layers.
Methods
The authors propose a purely weight-based approach using cosine similarity between a neuron's weight vectors:
- $\mathbf{w}_{in}$ (what it reads)
- $\mathbf{w}_{gate}$ (what determines its activation scaling)
- $\mathbf{w}_{out}$ (what it writes back)
To make these weights interpretable without changing model behavior, they introduce a preprocessing step called RefactorGLU. This leverages the symmetry property of gated activation functions to ensure that $\mathbf{w}_{gate}$ and $\mathbf{w}_{in}$ always have a non-negative cosine similarity.
Using these similarities, they propose an IO taxonomy (illustrated in Figure 1) to classify neurons into categories such as Strengthening (writing in the same direction as reading), Weakening (writing in the opposite direction), Conditional Strengthening/Weakening, and Proportional Change.
To evaluate the functional impact of these neuron classes, the authors perform causal ablations (zeroing or mean-substituting activations) and conditional ablations based on four distinct activation quadrants determined by the signs of the gate ($x_{gate}$) and input ($x_{in}$) scalar products.
Datasets
The experiments were conducted using:
- Dolma: A 20-million token random subset of the open corpus used to evaluate activation frequencies and run ablation experiments across models like OLMo, Gemma, and Llama.
- Wikipedia Dump (October 20, 2021): Used to construct a subject-attribute dataset to evaluate the "attribute rate" (a proxy for factual recall/subject enrichment).
Key Insights and Ideas
- The Strengthening-to-Weakening Progression: Across nine different LLMs, there is a striking architectural pattern: early-middle layers are dominated by conditional strengthening neurons, whereas late layers shift toward weakening functionality.
- The Rarity Paradox: Weakening neurons constitute a very small percentage of the network's total neurons, yet they activate far more frequently than strengthening neurons and have a disproportionately massive influence on the model's final outputs.
- Negative Gate Values Matter: In activation functions like Swish, negative gate values produce small, non-zero outputs. Mechanistic interpretability research often treats these as negligible noise. However, this paper reveals that these negative gate values actively drive model behavior, effectively flipping a weakening neuron into a strengthening mechanism.
Key Results
- High Frequency of Activation: The authors found a strong negative correlation (up to $-0.91$ for OLMo-7B) between $\cos(\mathbf{w}_{in}, \mathbf{w}_{out})$ and activation frequency, proving that weakening neurons are highly active.
- Outsize Ablation Impact: Mean-ablating the small class of weakening neurons caused a massive disruption to the model's output loss and entropy compared to ablating random baselines or other neuron classes.
- The Sharpening Mystery: Intriguingly, removing weakening neurons frequently caused the model's output distribution to sharpen rather than flatten. Conditional ablations traced this effect back to specific activation quadrants—particularly quadrant (iii), where both the gate and input values are negative.
Limitations and Future Work
- Single-Neuron Focus: The weight-cosine approach looks at individual neurons in isolation. It does not explicitly account for superposition, where features are distributed across sparse linear combinations of neurons (though the authors note their method can complement Sparse Autoencoders).
- Residual Connection Dependency: The framework relies on the assumption that representations across layers are directly comparable via the residual stream. It would not apply to newer, residual-free transformer architectures.
- Speculative Downstream Mechanisms: While the paper proves that weakening neurons have an outsize impact, the exact downstream mechanisms (e.g., whether they perform error correction or conflict resolution) remain compelling hypotheses that require deeper, circuit-level future work.
Summary updated