Large-scale Neuron Community Discovery Reveals Neuron-Sparse Behaviors in Language Models

Igor Shilov, Thomas Fel, Tal Haklay, Lucius Bushnaq, Owen LewisarXiv 2026 · arXiv

TL;DR

Researchers have developed an unsupervised method called "neuron community discovery" to identify small, reusable groups of neurons that perform specific tasks across millions of different prompts. By analyzing Llama-3.1-8B, they found that thousands of these sparse "communities" are causally responsible for behaviors ranging from basic grammar rules to complex multi-step arithmetic.

What Problem Does This Paper Address?

In the field of mechanistic interpretability, we want to understand the "basic units of computation" inside a Large Language Model (LLM). While we know that individual forward passes are sparse (only a few neurons fire strongly), it hasn't been clear if these active neurons form coherent, reusable "circuits" that the model recruits whenever it encounters a specific type of task.

Existing methods like Sparse Autoencoders (SAEs) try to reconstruct activations, but they don't always capture the actual computation being performed. Other studies focus on human-selected tasks (like "find the gendered pronoun circuit"), which limits discovery to what humans already expect to find. This paper addresses the need for a scalable, unsupervised way to find functional units of neurons that are consistently used across diverse, real-world data.

Methods

The authors introduce a four-step pipeline to discover and validate these neuron communities:

  1. Neuron Attribution: For 10 million prompts, they use a gradient-based score (activation $\times$ gradient) to identify which specific neurons are most influential for the model's top prediction.
  2. Co-occurrence Graph: They build a massive graph where neurons are nodes. An edge is drawn between two neurons if they are frequently "important" for the same prompts. They use z-scores to ensure they aren't just linking "popular" neurons that fire all the time.
  3. Community Detection: They apply the Leiden algorithm to this graph to find clusters (communities) of neurons that consistently work together.
  4. Causal Testing: To prove these communities actually do something, they run three tests:
  5. Necessity: If we turn off (ablate) these neurons, does the model's prediction break?
  6. Sufficiency: If we turn off everything else in those layers except this community, does the model still make the correct prediction?
  7. Layer Necessity: Does removing the entire set of layers break the prediction? (This ensures the community isn't just riding the coattails of a layer that does nothing).
Figure 1: Discovering and testing neuron communities. A: We compute per-prompt neuron attributions, then construct a co-occurrence graph, and run the Leiden community detection algorithm. B: We test community necessity and sufficiency (within a subset of layers) for predictions on held-out prompts.

Datasets

The study used a massive corpus of 20 million prompts (10M for discovery, 10M for evaluation) drawn from:

  • FineWeb-Edu: High-quality educational web pages.
  • English Wikipedia: General knowledge.
  • FineMath: Mathematical and technical content.
  • Targeted Slices: A curated 500k-prompt set for specific behaviors like code, grammar, and Q&A to ensure they captured rare but important behaviors.

Key Insights and Ideas

  • Redundant Voting: Many communities operate like a committee. In the "an + vowel" agreement community, removing any single neuron does almost no damage, but removing the whole group causes the model to fail. This suggests the model builds in redundancy for critical tasks.
  • Cross-Layer Sparsity: These functional units aren't confined to a single layer. Some communities span up to 15 layers, showing that sparse computation is a "vertical" phenomenon in the model's architecture.
  • Functional Diversity: The method discovered communities for highly specific tasks: suppressing the word "fraction," completing LaTeX commands, handling plural morphology (-s), and even performing prime factorization.
  • Native Neurons Matter: While much recent research focuses on SAE features, this paper shows that native MLP neurons themselves form interpretable, sparse circuits that are highly faithful to the model's actual behavior.

Key Results

  • Coverage: 10.3% of all held-out prompts could have at least one layer's contribution explained by a sparse community of 128 neurons or fewer.
  • Grammar Case Study: They identified a 6-neuron community (across 6 layers) responsible for "an + vowel" agreement.
  • Necessity: Ablating these 6 neurons changed the prediction significantly (median KL divergence of 4.99).
  • Sufficiency: Keeping only these 6 neurons (out of ~86,000 in those layers) preserved the prediction almost perfectly (median KL of 0.053).
  • Taxonomy: Using an LLM to classify the discovered communities, they found that "Common Phrases" (42.3%) and "Token Suppression" (19.5%) were the most frequent, but "Grammar" and "Numerical Operations" were also significant.
A "deep dive" into the redundancy of the an + vowel community, showing how different subsets of neurons contribute to the final prediction.

Limitations and Future Work

  • Final Token Only: The current method only looks at the very last token of a prompt. It likely misses communities that perform logic across multiple tokens (e.g., reading a math problem and then solving it).
  • Non-Overlapping Groups: The Leiden algorithm forces each neuron into exactly one community per run. In reality, a neuron might be "polysemantic" and participate in multiple different circuits.
  • The 90% Gap: While 10% coverage is a massive step for unsupervised interpretability, 90% of prompts remain unexplained. This could be because some computations are truly dense (not sparse) or because the discovery data didn't contain enough examples to link the relevant neurons.
  • Future Direction: The authors suggest moving toward "cross-position" community discovery to capture more complex, multi-step reasoning.

Summary updated

Abstract

What reusable units of computation can we discover within a language model's neurons? Prior work has shown that individual forward passes can often be sparsified post-hoc to small groups of neurons, but it has been unclear whether these groups form coherent computational entities reused across inputs. We introduce neuron community discovery, an unsupervised method for finding neuron groups that recur across prompts and are causally necessary and sufficient, within the layers they span, for common computations. The method combines gradient-based attribution, a graph connecting neurons that are repeatedly attributed together, and community detection. Applied to Llama-3.1-8B using ten million prompts, it discovers thousands of candidate communities. We evaluate them on held-out prompts through three causal tests: community necessity, sufficiency within a layer subset, and layer necessity. These tests identify groups whose removal disrupts predictions and whose retention preserves predictions despite ablating the remaining MLP neurons in the tested layers. Discovered communities support functions like grammatical agreement, token-probability calibration, fixed phrases, in-context copying, addition, and multiplication. With communities capped at 128 neurons across all layers, 10.3% of held-out prompts have at least one layer's contribution explained on a neuron-sparse basis. Coverage remains 8.7% when requiring at least four layers and 3.3% when requiring at least twelve. These results show that small groups of existing neurons can provide reusable computational components across prompts, and that such groups can be discovered at scale.
BibTeX
@misc{shilov2026large,
  author = {Igor Shilov and Thomas Fel and Tal Haklay and Lucius Bushnaq and Owen Lewis},
  title = {{Large-scale Neuron Community Discovery Reveals Neuron-Sparse Behaviors in Language Models}},
  year = {2026},
  eprint = {2611.23112},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.23112},
}
Figure