Introduction
In Natural Language Processing, Transformer models are often characterized by their "distributed" representations, where information is assumed to be spread across high-dimensional vectors and the entire input sequence via self-attention. This paper challenges that assumption by examining a specific grammatical task: French object-past participle agreement. The study investigates whether the syntactic information required for this task is truly distributed or if it is localized in a manner reflecting human linguistic structures.
The Agreement Task
The researchers focused on sentences like "les amis que j'ai rencontrés" (the friends that I met). In French, the past participle rencontrés must agree in number with the preceding object amis. To perform this correctly, a model must track the "plural" feature from the antecedent noun, through the relative pronoun que, to the final verb. This provides a clear trace for observing how and where the model stores grammatical information.
Methodology
The study employed three primary techniques to interrogate an incremental Transformer model trained on French Wikipedia:
- Probing: Training logistic regression classifiers to predict the target verb's number from internal word representations.
- Causal Intervention: Selectively masking tokens during self-attention to identify which words are functional for the model's decision.
- Feature Selection: Using $\ell_1$ regularization (Lasso) to determine if information is concentrated in specific dimensions of the 768-dimensional word vectors.
Locality of Syntactic Information
A key finding is that syntactic information follows a "just-in-time" delivery pattern. Probing accuracy is low in the "prefix" (tokens before the antecedent), peaks sharply in the "context" (the gap between the noun and the verb), and drops off immediately in the "suffix" (tokens after the verb). This suggests the model "loads" the necessary number information only when the grammatical dependency is active and discards it once the requirement is satisfied.
Causal analysis further revealed that the model relies most heavily on the antecedent and the relative pronoun que, aligning with formal linguistic theory. Even in the presence of "attractor" nouns (nouns with a different number intended to confuse the model), the model maintains high accuracy by focusing on these specific local tokens.
Sparsity and Redundancy
While the model's representations are high-dimensional, the relevant syntactic signal is remarkably sparse. Feature selection experiments demonstrated that a probe using only 90 out of 768 dimensions can achieve over 90% accuracy, compared to 94.8% for the full vector. This indicates that the information is concentrated in a small fraction of the available "neurons."
Interestingly, the model also exhibits redundancy. If the top 90 dimensions are removed, the model can still recover the information from the remaining dimensions, albeit with a slight performance penalty. This suggests a fail-safe mechanism where information is primary in a few dimensions but backed up across others.
Conclusion
This research demonstrates that for specific syntactic tasks, Transformer representations are less "distributed" than commonly believed. Information is localized both temporally (within specific tokens in the sequence) and spatially (within a small subset of vector dimensions). These findings suggest that Transformers may learn internal strategies that closely mirror the localized, hierarchical nature of human syntax.
Summary updated