Weakening Neurons: An Input-Output Functionality in Transformers with Outsize Influence

Sebastian Gerstner, Hilal AlQuabeh, Kentaro Inui, Hinrich SchützearXiv 2026 · arXiv

TL;DR

This paper introduces a weight-based taxonomy to analyze how gated neurons in Large Language Models (LLMs) read from and write to the residual stream. It discovers a small but highly influential class of "weakening neurons" in late layers that actively suppress specific informational directions, exerting an outsize impact on model behavior even during negative gating states.

What Problem Does This Paper Address?

Mechanistic interpretability aims to reverse-engineer the inner workings of transformer models. While previous research has categorized neurons based on when they activate or what tokens they promote, it often overlooks the direct relationship between what a neuron reads from the residual stream and what it writes back.

This input-output (IO) relationship is especially complex in modern LLMs (like Llama, Gemma, and OLMo) that use Gated Linear Units (GLUs) such as SwiGLU or GEGLU. Because GLU neurons involve three weight vectors (input, gate, and output), understanding their joint functionality requires a more sophisticated framework than traditional ReLU networks. This paper addresses this gap by establishing a formal IO taxonomy for gated neurons to understand how they manipulate representations across different layers.

Methods

The authors propose a purely weight-based approach using cosine similarity between a neuron's weight vectors:

  • $\mathbf{w}_{in}$ (what it reads)
  • $\mathbf{w}_{gate}$ (what determines its activation scaling)
  • $\mathbf{w}_{out}$ (what it writes back)

To make these weights interpretable without changing model behavior, they introduce a preprocessing step called RefactorGLU. This leverages the symmetry property of gated activation functions to ensure that $\mathbf{w}_{gate}$ and $\mathbf{w}_{in}$ always have a non-negative cosine similarity.

Using these similarities, they propose an IO taxonomy (illustrated in Figure 1) to classify neurons into categories such as Strengthening (writing in the same direction as reading), Weakening (writing in the opposite direction), Conditional Strengthening/Weakening, and Proportional Change.

Figure 1: Our IO taxonomy for gated neurons. We classify neurons by how their input, gate, and output weight directions align. See Section 4.2.2 for details.

To evaluate the functional impact of these neuron classes, the authors perform causal ablations (zeroing or mean-substituting activations) and conditional ablations based on four distinct activation quadrants determined by the signs of the gate ($x_{gate}$) and input ($x_{in}$) scalar products.

Datasets

The experiments were conducted using:

  • Dolma: A 20-million token random subset of the open corpus used to evaluate activation frequencies and run ablation experiments across models like OLMo, Gemma, and Llama.
  • Wikipedia Dump (October 20, 2021): Used to construct a subject-attribute dataset to evaluate the "attribute rate" (a proxy for factual recall/subject enrichment).

Key Insights and Ideas

  • The Strengthening-to-Weakening Progression: Across nine different LLMs, there is a striking architectural pattern: early-middle layers are dominated by conditional strengthening neurons, whereas late layers shift toward weakening functionality.
  • The Rarity Paradox: Weakening neurons constitute a very small percentage of the network's total neurons, yet they activate far more frequently than strengthening neurons and have a disproportionately massive influence on the model's final outputs.
  • Negative Gate Values Matter: In activation functions like Swish, negative gate values produce small, non-zero outputs. Mechanistic interpretability research often treats these as negligible noise. However, this paper reveals that these negative gate values actively drive model behavior, effectively flipping a weakening neuron into a strengthening mechanism.

Key Results

  • High Frequency of Activation: The authors found a strong negative correlation (up to $-0.91$ for OLMo-7B) between $\cos(\mathbf{w}_{in}, \mathbf{w}_{out})$ and activation frequency, proving that weakening neurons are highly active.
Figure 5: Heatmap of cos(win, wout) (x-axis) vs. activa- tion frequency (y-axis), when running OLMo-7B on a subset of Dolma (Section 6). The darker the color, the more neurons in the given area. This result is evidence that weakening neurons are a disproportionately impor- tant IO functionality, despite being only a small class.
  • Outsize Ablation Impact: Mean-ablating the small class of weakening neurons caused a massive disruption to the model's output loss and entropy compared to ablating random baselines or other neuron classes.
  • The Sharpening Mystery: Intriguingly, removing weakening neurons frequently caused the model's output distribution to sharpen rather than flatten. Conditional ablations traced this effect back to specific activation quadrants—particularly quadrant (iii), where both the gate and input values are negative.
Effect of conditional ablations on model output entropy

Limitations and Future Work

  • Single-Neuron Focus: The weight-cosine approach looks at individual neurons in isolation. It does not explicitly account for superposition, where features are distributed across sparse linear combinations of neurons (though the authors note their method can complement Sparse Autoencoders).
  • Residual Connection Dependency: The framework relies on the assumption that representations across layers are directly comparable via the residual stream. It would not apply to newer, residual-free transformer architectures.
  • Speculative Downstream Mechanisms: While the paper proves that weakening neurons have an outsize impact, the exact downstream mechanisms (e.g., whether they perform error correction or conflict resolution) remain compelling hypotheses that require deeper, circuit-level future work.

Summary updated

Abstract

We analyze the learned input-output behavior of GLU-based neurons in large language models (LLMs). We propose a simple analysis method: For each neuron, we compute the cosine similarities between its input (reading) and output (writing) weight vectors. In this scheme, a strong negative cosine similarity indicates the neuron weakens the direction it detects in the residual stream, so we call this a weakening neuron. This allows us to gain a number of novel insights. First, we show that nine different LLMs have similar patterns: weakening neurons appear mostly in late layers whereas their counterparts, (conditional) strengthening neurons, are frequent in early-middle layers. Second, we find that weakening neurons display surprising behavior: even though there are few, they activate often and have a large influence on model behavior. Third, weakening neurons have a strong effect on model output when gate values are negative -- which is surprising since negative gate values are not expected to encode functionality.
BibTeX
@misc{gerstner2026weakening,
  author = {Sebastian Gerstner and Hilal AlQuabeh and Kentaro Inui and Hinrich Schütze},
  title = {{Weakening Neurons: An Input-Output Functionality in Transformers with Outsize Influence}},
  year = {2026},
  eprint = {2609.18612},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2609.18612},
}
Figure