How Distributed are Distributed Representations? An Observation on the Locality of Syntactic Information in Verb Agreement Tasks

Bingzhi Li, Guillaume Wisniewski, Benoît CrabbéACL 2022 · DOI

Introduction

In Natural Language Processing, Transformer models are often characterized by their "distributed" representations, where information is assumed to be spread across high-dimensional vectors and the entire input sequence via self-attention. This paper challenges that assumption by examining a specific grammatical task: French object-past participle agreement. The study investigates whether the syntactic information required for this task is truly distributed or if it is localized in a manner reflecting human linguistic structures.

The Agreement Task

The researchers focused on sentences like "les amis que j'ai rencontrés" (the friends that I met). In French, the past participle rencontrés must agree in number with the preceding object amis. To perform this correctly, a model must track the "plural" feature from the antecedent noun, through the relative pronoun que, to the final verb. This provides a clear trace for observing how and where the model stores grammatical information.

Methodology

The study employed three primary techniques to interrogate an incremental Transformer model trained on French Wikipedia:

  1. Probing: Training logistic regression classifiers to predict the target verb's number from internal word representations.
  2. Causal Intervention: Selectively masking tokens during self-attention to identify which words are functional for the model's decision.
  3. Feature Selection: Using $\ell_1$ regularization (Lasso) to determine if information is concentrated in specific dimensions of the 768-dimensional word vectors.

Locality of Syntactic Information

A key finding is that syntactic information follows a "just-in-time" delivery pattern. Probing accuracy is low in the "prefix" (tokens before the antecedent), peaks sharply in the "context" (the gap between the noun and the verb), and drops off immediately in the "suffix" (tokens after the verb). This suggests the model "loads" the necessary number information only when the grammatical dependency is active and discards it once the requirement is satisfied.

Probing accuracy across token positions shows a clear peak in the context between the antecedent and the verb.

Causal analysis further revealed that the model relies most heavily on the antecedent and the relative pronoun que, aligning with formal linguistic theory. Even in the presence of "attractor" nouns (nouns with a different number intended to confuse the model), the model maintains high accuracy by focusing on these specific local tokens.

Sparsity and Redundancy

While the model's representations are high-dimensional, the relevant syntactic signal is remarkably sparse. Feature selection experiments demonstrated that a probe using only 90 out of 768 dimensions can achieve over 90% accuracy, compared to 94.8% for the full vector. This indicates that the information is concentrated in a small fraction of the available "neurons."

Probing accuracy as a function of the number of non-zero dimensions, showing rapid saturation around 90-100 dimensions.

Interestingly, the model also exhibits redundancy. If the top 90 dimensions are removed, the model can still recover the information from the remaining dimensions, albeit with a slight performance penalty. This suggests a fail-safe mechanism where information is primary in a few dimensions but backed up across others.

Conclusion

This research demonstrates that for specific syntactic tasks, Transformer representations are less "distributed" than commonly believed. Information is localized both temporally (within specific tokens in the sequence) and spatially (within a small subset of vector dimensions). These findings suggest that Transformers may learn internal strategies that closely mirror the localized, hierarchical nature of human syntax.

Summary updated

BibTeX
@inproceedings{li2022how,
  author = {Bingzhi Li and Guillaume Wisniewski and Benoît Crabbé},
  title = {{How Distributed are Distributed Representations? An Observation on the Locality of Syntactic Information in Verb Agreement Tasks}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2022},
  doi = {10.18653/v1/2022.acl-short.54},
}
Figure