Understanding Gated Neurons in Transformers from Their Input-Output Functionality

Sebastian Gerstner, Hinrich SchützearXiv 2025 · DOI · arXiv

Abstract

Modern Transformer-based Large Language Models (LLMs) rely on a residual stream to communicate information across layers. While individual neurons within Multi-Layer Perceptron (MLP) blocks are known to be critical, their specific functional roles remain poorly understood. This paper investigates the relationship between a neuron's input and output weights in gated activation functions (e.g., SwiGLU). By analyzing 12 major LLMs, we identify a systematic "division of labor" where early layers primarily perform "enrichment" of concepts, while final layers engage in "residual sharpening" through "depletion." This parameter-based approach allows for a functional classification of 100% of the neurons in a model.

Introduction

In a Transformer architecture, the residual stream acts as a shared workspace where layers read and write information. MLP neurons are the primary units performing these operations. Previous interpretability research has often decoupled the "input side" (activation context) from the "output side" (contribution to the stream). This study addresses this gap by asking: What is the relationship between what a neuron reads and what it writes? Understanding whether a neuron amplifies a detected concept (enrichment) or removes it (depletion) provides a clearer picture of how Transformers process information.

Methodology

The authors focus on gated activation functions, which utilize three weight vectors: a gate ($w_{gate}$), a linear input ($w_{in}$), and an output ($w_{out}$). By calculating the cosine similarity between these vectors, they categorize neurons into a novel taxonomy.

Taxonomy of gated neurons based on cosine similarity between gate, input, and output weight vectors.

The primary categories include:

  • Enrichment: The neuron detects a direction and adds more of it to the residual stream.
  • Depletion: The neuron detects a direction and reduces its presence in the stream.
  • Conditional Enrichment/Depletion: The operation is contingent on a separate gating condition.
  • Proportional Change: Modifies one concept based on the presence of another.
  • Orthogonal Output: Writes to a direction unrelated to what it detected.

This classification was validated using the "logit lens" to project weights into vocabulary space and by analyzing activations across 20 million tokens from the Dolma dataset.

Results

The study analyzed 12 models, including the Llama, Gemma, OLMo, Mistral, Qwen, and Yi families. Several universal patterns emerged:

The Layer-Wise Shift

A consistent "stages of inference" pattern was observed across all models. Early-to-middle layers are dominated by "enrichment" neurons that build up and refine feature representations. As the model approaches the final layers, there is a significant shift toward "depletion" neurons.

Median cosine similarity between input and output weights across network depth for 12 LLMs, showing the transition from enrichment to depletion.

This "residual sharpening" suggests that the model actively prunes unnecessary information to clarify the final token prediction. In Llama models, enrichment peaks between layers 7 and 11, while the final layers consistently show negative cosine similarity, indicating depletion.

High-Precision Gating

The researchers discovered a "double checking" phenomenon where $w_{gate}$ and $w_{in}$ are mathematically orthogonal but semantically identical. This configuration allows the neuron to act as a high-precision "AND" gate, firing only when a concept is detected with extreme certainty.

Discussion and Limitations

While this parameter-based approach provides a functional classification for 100% of neurons, several limitations remain. First, the analysis focuses on neurons in isolation, potentially missing group dynamics such as "superposition." Second, while cosine similarity is a strong geometric proxy, it does not guarantee semantic identity in all cases. Finally, the causal role of these neurons in specific inference tasks (e.g., factual recall) requires further validation through ablation studies.

Conclusion

By examining the interaction between input and output weights, this work uncovers a systematic functional organization within Transformers. The transition from enrichment to depletion across layers provides a new lens through which to view the internal "thinking" process of Large Language Models.

Summary updated

BibTeX
@misc{gerstner2025understanding,
  author = {Sebastian Gerstner and Hinrich Schütze},
  title = {{Understanding Gated Neurons in Transformers from Their Input-Output Functionality}},
  year = {2025},
  eprint = {2505.17936},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2505.17936},
}
Figure