Fine-Grained Mechanistic Interpretability at the Neuron Level

32 concepts31 papers1 proposal

For years, deep learning models were treated as "black boxes" where knowledge was thought to be inextricably tangled within high-dimensional vector spaces. While researchers could identify which layers were active during a task, the internal logic remained opaque. However, a growing body of research suggests that this "monolithic" view is a limitation of our tools rather than the models themselves. By shifting focus from entire layers to individual neurons and fine-grained circuits, we can begin to see the precise computational units that drive model behavior.

The Granularity Gap

Traditional interpretability often relies on "probing"—training a classifier to see if a layer contains certain information. While useful, Geiger et al. argue that analyzing internal dynamics at the level of entire vector representations provides only a coarse view of logic. This lack of resolution makes it difficult to determine if a model has truly learned a concept or is simply performing sophisticated pattern matching.

Dalvi et al. illustrate this by asking: "What is one grain of sand in the desert?" They found that while linguistic knowledge might seem distributed, it is often anchored by a tiny fraction of "important" neurons. In fact, zeroing out just a small percentage of these top-ranked neurons can cripple a model's performance on specific tasks, while removing half of the least important neurons has almost no effect. This suggests that models are far sparser and more localized than their dense architectures imply.

Localizing Functional Units

If knowledge is localized, can we find the specific "playmakers" responsible for it? Mohebbi et al. discovered that models like BERT often anchor linguistic information in specific tokens—such as using the ##s suffix to track grammatical number.

Beyond tokens, researchers have identified individual neurons dedicated to specific linguistic tasks. Bau et al. demonstrated that by identifying neurons correlated with properties like verb tense or gender, they could "steer" a model's output. By manually tweaking the activations of just a few neurons, they could force a translation model to change the gender of a subject, proving these units are not just correlated with features but are causally responsible for them.

This granularity extends to complex reasoning as well. Stolfo et al. used causal mediation analysis to move beyond treating attention layers as single units, instead isolating the specific attention heads responsible for routing information between operands in arithmetic problems.

The Fragility of Sparse Mechanisms

Understanding the model at a granular level reveals why certain behaviors are so difficult to control. Wei et al. found that safety alignment in Large Language Models (LLMs) is surprisingly fragile because it relies on a very sparse subset of weights—roughly 3% of neurons. Because this "safety circuit" is so small, it can be surgically removed or bypassed without damaging the model's general utility, explaining why jailbreaking remains a persistent challenge.

Similarly, Liu showed that a model's sense of confidence is not a separate module but is embedded within the same circuits used to generate facts. When the specific neurons representing a fact are disrupted, the model's internal representation of certainty about that fact breaks simultaneously.

Mapping the Circuitry

The ultimate goal of this research direction is to move from identifying "bricks" to understanding the "architecture." Yao et al. argue that looking at isolated components is insufficient; we must identify "knowledge circuits"—interconnected groups of neurons and heads that work together to retrieve and process information.

To facilitate this, tools like the NeuroX library (Dalvi et al.) have been developed to help researchers rank and visualize individual neurons. However, as Gupta et al. note, even our current "ground-truth" circuits might be too coarse. They propose that the field must transition toward feature-level benchmarks—using techniques like sparse autoencoders—to align our interpretations with the true, fine-grained resolution at which these models operate.

Key Foundational Work

Identifying and Controlling Important Neurons in Neural Machine Translation (2019) — contributed 1 concept elements to this direction.
What Is One Grain of Sand in the Desert? Analyzing Individual Neurons in Deep NLP Models (2019) — contributed 1 concept elements to this direction.
Neural Natural Language Inference Models Partially Embed Theories of Lexical Entailment and Negation (2020) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_neuron_level_interpretability_39299ff7,
  title = {neuron-level interpretability},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/neuron-level-interpretability/rdr_39299ff73f804ab68897d10e7920f453},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_neuron_level_interpretability_39299ff7_v1,
  title = {neuron-level interpretability},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/neuron-level-interpretability/rdr_39299ff73f804ab68897d10e7920f453/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake