TL;DR
Researchers used Sparse Autoencoders (SAEs) to map the internal "circuits" LLMs use to process English subject-verb agreement. They discovered that models treat the singular as a "default" state and only use specific internal "symbols" to mark plurality—a finding that mirrors human linguistic theories and allows for surgical edits of a model's grammatical choices.
What Problem Does This Paper Address?
Large Language Models (LLMs) are famously "black boxes." While they exhibit near-perfect grammar, we don't fully understand how they represent rules like subject-verb agreement (e.g., why they choose "the men are" instead of "the men is").
The challenge lies in polysemanticity: a single "neuron" in a model might activate for both "lipstick" and "wild boars," making it nearly impossible for humans to read the model's "thoughts" directly from its hidden states. This paper aims to untangle these states to find the specific "symbols" the model uses to track grammatical number and propagate that information from a subject noun to a verb.
Methods
The authors employ Sparse Autoencoders (SAEs), a popular tool in mechanistic interpretability. SAEs act like a prism, decomposing the model's dense, indecipherable hidden vectors into a massive list of sparse, human-interpretable "features" or "symbols."
Key methodological innovations include:
- Matryoshka SAEs: A multi-level training approach that allows for feature extraction at different scales.
- Behaviorist Framework for "Meaning": Instead of guessing what a feature does, the authors define a feature's "meaning" by its causal effect: if adding a feature to a hidden state forces the model to output a plural verb, that feature "means" plurality.
- Singular Conjugation Preference Metric ($B$): A custom function that calculates the log-probability ratio of singular vs. plural verb forms across hundreds of verb pairs, allowing for a precise scalar measurement of the model's grammatical "intent."
- Activation Patching: A "surgical" technique where they manually insert or remove specific SAE features from the model's internal processing to observe changes in the final output.
Datasets
The study used several syntactically diverse datasets to test different types of agreement:
- nounpp: Sentences with prepositional phrase distractors (e.g., "The man in the park...").
- rc & long_rc: Sentences with relative clauses (e.g., "The man that the dog sees...").
- within_rc: Agreement within an embedded clause.
- OpenWebText: Used for the initial training of the SAEs.
- Custom Verb/Noun Pairs: They identified 367 monotokenal verb pairs (is/are, has/have) and 1,700 noun pairs to ensure results weren't biased by a single word choice.
Key Insights and Ideas
- The "Unmarked" Singular: Perhaps the most striking finding is that LLMs (Pythia, Gemma, and Llama) don't seem to have a "singular" feature. Instead, they use a privative representation: the model defaults to singular, and "plurality" is a specific tag added on top. If you strip away the "plural" features, the model immediately reverts to singular. This aligns with the linguistic theory of "markedness."
- Translinguistic Symbols: Some internal features are "polyglot." A feature that triggers the English plural "are" also increases the probability of plural verbs in French (sont), Italian (hanno), and Portuguese (são).
- Information "Spraying": In complex sentences, the model "sprays" plural information across multiple tokens in a clause. This explains "number distraction" errors (e.g., "The key to the cabinets are..."); the plural signal from "cabinets" interferes with the singular signal from "key."
- The 2-Hop Circuit: The authors mapped how information "hops" from the subject to the verb. They found specific "sluice gate" features in the attention heads that open up to allow the plural symbol to flow from the noun's residual stream to the final token where the verb is generated.
Key Results
- High Interpretability: The SAEs explained 80-90% of the variance in the models' hidden states, proving they are effective tools for "reading" the model.
- Causal Success: By intervening and "patching" just a few features (like
0:mlp:134in Pythia-70m), the researchers could consistently flip the model's output from singular to plural and vice versa. - Cross-Model Consistency: While the specific feature numbers differed, the "plural-as-marked/singular-as-default" logic held true across Pythia-70M, Gemma-2-2b, and Llama-3.1-8b.
- Semantic Clustering: Plural features weren't just generic; they often had semantic "flavors." For example, one feature might specifically track animate plurals (men, women, children), while another tracks inanimate objects.
Limitations and Future Work
- Feature Splitting: The authors suspect SAEs might "spuriously split" a single concept into multiple redundant features, making the "circuit diagrams" more complex than necessary.
- Gradient Shrinking: The gradient-based method used to find features in smaller models struggled with larger models like Llama-3.1-8b, requiring more expensive "brute-force" calculations.
- Multi-token Verbs: The current metric ($B$) only works for verbs that are a single "token." Understanding how models handle multi-token verbs (like "uphold") remains an open challenge.
- The "Why" of Defaults: While they found that the model defaults to singular, they haven't yet identified the specific mechanism that sets this default in the first place.
Summary updated