TL;DR
Language models learn subject-verb agreement (SVA) remarkably early in training, but the internal machinery responsible for it continues to evolve long after performance plateaus. This study reveals that SVA circuits undergo a process of "mechanistic sparsification," shifting from a diffuse collection of neurons to a strikingly compact and efficient circuit—sometimes involving as few as 18 neurons and a single attention head.
What Problem Does This Paper Address?
While we know that large language models (LLMs) excel at syntactic tasks like subject-verb agreement (e.g., knowing that "The keys to the cabinet are..." requires a plural verb), we don't fully understand how this competence emerges during training.
A central question in mechanistic interpretability is whether a model's internal "algorithm" is fixed once it achieves high accuracy, or if the underlying hardware—the specific neurons and attention heads—continues to reorganize. The authors investigate this by tracking the "circuit" for SVA across the entire training trajectory of the Pythia model family. They aim to move beyond just measuring when a model becomes accurate to understanding how its internal representation of linguistic rules matures.
Methods
The researchers analyzed pythia-1b and ten different random seeds of pythia-410m. They focused on a "minimal pair" task: comparing the model's preference for singular vs. plural verbs given singular vs. plural subjects, often with intervening "distractor" phrases.
To find the circuits, they employed several state-of-the-art interpretability techniques:
- Relevance Patching (RelP): A gradient-based attribution method used to identify which MLP neurons contribute most to the SVA task.
- Causal Interventions: They measured Necessity (does performance crash if we "turn off" these components?) and Sufficiency (can these components alone drive the behavior if we swap them into a different context?).
- Path Patching: Used to determine the "routing" of information—specifically, which neurons write to which attention heads.
- Distributed Alignment Search (DAS): A method to find if high-dimensional information (like grammatical number) is actually compressed into a low-dimensional (1D) subspace within the model's activations.
Datasets
The study utilized several datasets for evaluation and stress-testing:
- Synthetic Templates: Custom-generated sentences featuring different syntactic distractors: Prepositional Phrases (PP), Subject Relative Clauses (SUBJECTRC), and Object Relative Clauses (OBJECTRC).
- WikiText: Naturalistic examples mined from Wikipedia to ensure the discovered circuits weren't just artifacts of synthetic data.
- Perturbed Templates: Variations involving adjective/adverb insertions and different prefixes to test the circuit's robustness to sentence structure changes.
Key Insights and Ideas
- Behavior-Mechanism Decoupling: Accuracy on SVA reaches near-peak levels within the first 2% of training (around step 2k). However, the internal circuit continues to change drastically until the end of training (step 143k).
- The Sparsification Trend: Early in training, the "knowledge" is spread across a diffuse set of neurons. As training progresses, the model "prunes" its internal solution, concentrating the task into a much smaller, more specialized set of neurons.
- Attention Head Rerouting: In pythia-1b, the model initially uses two specific attention heads in layer 15 to transport "number" information. Later in training, it shifts primary responsibility to a completely different head (L13 H7).
- 1D Causal Channels: The authors found that the entire signal for "singular vs. plural" carried by the primary attention head can be localized to a single dimension. This suggests the model learns to highly compress abstract syntactic features.
Key Results
- Extreme Sparsity: For the final pythia-1b model, a hand-constructed circuit of just 18 neurons and 1 attention head captured nearly the entire agreement signal. This tiny subset was 78% necessary and 83% sufficient for the task.
- Generalization: This sparse circuit wasn't overfit; it generalized successfully to natural text from WikiText and to complex sentence structures it wasn't "discovered" on.
- Consistency Across Seeds: While the specific heads used varied between different random seeds of the 410m model, the pattern of sparsification was universal.
Limitations and Future Work
- Scope: The study is limited to one specific syntactic task (SVA) and one language (English). It remains to be seen if more complex semantic or reasoning tasks undergo similar sparsification.
- Manual Refinement: While the initial neuron identification was automated, the final 18-neuron circuit involved some manual "judgment calls" and iterative pruning. Developing fully automated methods for discovering such sparse, generalizable circuits is a key next step.
- The "Why": The paper documents that sparsification happens but doesn't definitively explain the pressure driving it. The authors hypothesize that diffuse circuits serve as a "scaffold" for the eventual emergence of more efficient, specialized ones—a process reminiscent of "grokking."
Summary updated