The Emergence and Sparsification of a Syntactic Circuit

Qing Yao, Sasha Boguraev, Tiago Pimentel, Kyle MahowaldarXiv 2026 · arXiv

TL;DR

Language models learn subject-verb agreement (SVA) remarkably early in training, but the internal machinery responsible for it continues to evolve long after performance plateaus. This study reveals that SVA circuits undergo a process of "mechanistic sparsification," shifting from a diffuse collection of neurons to a strikingly compact and efficient circuit—sometimes involving as few as 18 neurons and a single attention head.

What Problem Does This Paper Address?

While we know that large language models (LLMs) excel at syntactic tasks like subject-verb agreement (e.g., knowing that "The keys to the cabinet are..." requires a plural verb), we don't fully understand how this competence emerges during training.

A central question in mechanistic interpretability is whether a model's internal "algorithm" is fixed once it achieves high accuracy, or if the underlying hardware—the specific neurons and attention heads—continues to reorganize. The authors investigate this by tracking the "circuit" for SVA across the entire training trajectory of the Pythia model family. They aim to move beyond just measuring when a model becomes accurate to understanding how its internal representation of linguistic rules matures.

Methods

The researchers analyzed pythia-1b and ten different random seeds of pythia-410m. They focused on a "minimal pair" task: comparing the model's preference for singular vs. plural verbs given singular vs. plural subjects, often with intervening "distractor" phrases.

To find the circuits, they employed several state-of-the-art interpretability techniques:

  • Relevance Patching (RelP): A gradient-based attribution method used to identify which MLP neurons contribute most to the SVA task.
  • Causal Interventions: They measured Necessity (does performance crash if we "turn off" these components?) and Sufficiency (can these components alone drive the behavior if we swap them into a different context?).
  • Path Patching: Used to determine the "routing" of information—specifically, which neurons write to which attention heads.
  • Distributed Alignment Search (DAS): A method to find if high-dimensional information (like grammatical number) is actually compressed into a low-dimensional (1D) subspace within the model's activations.

Datasets

The study utilized several datasets for evaluation and stress-testing:

  • Synthetic Templates: Custom-generated sentences featuring different syntactic distractors: Prepositional Phrases (PP), Subject Relative Clauses (SUBJECTRC), and Object Relative Clauses (OBJECTRC).
  • WikiText: Naturalistic examples mined from Wikipedia to ensure the discovered circuits weren't just artifacts of synthetic data.
  • Perturbed Templates: Variations involving adjective/adverb insertions and different prefixes to test the circuit's robustness to sentence structure changes.

Key Insights and Ideas

  • Behavior-Mechanism Decoupling: Accuracy on SVA reaches near-peak levels within the first 2% of training (around step 2k). However, the internal circuit continues to change drastically until the end of training (step 143k).
  • The Sparsification Trend: Early in training, the "knowledge" is spread across a diffuse set of neurons. As training progresses, the model "prunes" its internal solution, concentrating the task into a much smaller, more specialized set of neurons.
  • Attention Head Rerouting: In pythia-1b, the model initially uses two specific attention heads in layer 15 to transport "number" information. Later in training, it shifts primary responsibility to a completely different head (L13 H7).
  • 1D Causal Channels: The authors found that the entire signal for "singular vs. plural" carried by the primary attention head can be localized to a single dimension. This suggests the model learns to highly compress abstract syntactic features.
The SVA circuit for pythia-1b at step 2k (left) is diffuse, while at step 143k (right) it is strikingly sparse.

Key Results

  • Extreme Sparsity: For the final pythia-1b model, a hand-constructed circuit of just 18 neurons and 1 attention head captured nearly the entire agreement signal. This tiny subset was 78% necessary and 83% sufficient for the task.
  • Generalization: This sparse circuit wasn't overfit; it generalized successfully to natural text from WikiText and to complex sentence structures it wasn't "discovered" on.
  • Consistency Across Seeds: While the specific heads used varied between different random seeds of the 410m model, the pattern of sparsification was universal.
Accuracy vs. the number of neurons (k50) shows that while accuracy stays high, the number of neurons needed to maintain that accuracy drops significantly over time.

Limitations and Future Work

  • Scope: The study is limited to one specific syntactic task (SVA) and one language (English). It remains to be seen if more complex semantic or reasoning tasks undergo similar sparsification.
  • Manual Refinement: While the initial neuron identification was automated, the final 18-neuron circuit involved some manual "judgment calls" and iterative pruning. Developing fully automated methods for discovering such sparse, generalizable circuits is a key next step.
  • The "Why": The paper documents that sparsification happens but doesn't definitively explain the pressure driving it. The authors hypothesize that diffuse circuits serve as a "scaffold" for the eventual emergence of more efficient, specialized ones—a process reminiscent of "grokking."

Summary updated

Abstract

To understand the emergence of grammatical behavior in language models, we track circuits for subject-verb number agreement in pythia-1b and ten seeds of pythia-410m across training. We find that subject-verb agreement is learned early, and is initially handled by a diffuse collection of neurons. However, after agreement behavior has stabilized, the mechanisms responsible for it continue to reorganize. The attention heads which mediate agreement often shift during training: in the case of pythia-1b, behavior is controlled by two heads early in training but by a third (different) head later. Furthermore, the circuit shrinks over training, with pythia-1b's final circuit being strikingly sparse. We take these results to be informative about how linguistic knowledge emerges and is represented in a language model.
BibTeX
@misc{yao2026emergence,
  author = {Qing Yao and Sasha Boguraev and Tiago Pimentel and Kyle Mahowald},
  title = {{The Emergence and Sparsification of a Syntactic Circuit}},
  year = {2026},
  eprint = {2610.33299},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.33299},
}
Figure