Probing for Referential Information in Language Models

Ionut-Teodor Sorodoc, Kristina Gulordava, Gemma BoledaACL 2020 · DOI

Introduction

Modern neural language models (LMs) are remarkably adept at predicting the next word in a sequence, often "accidentally" learning complex syntax like subject-verb agreement. However, a fundamental question persists: do these models truly understand reference? In linguistics, coreference involves linking disparate mentions—such as "the president," "he," and "Mr. Korotich"—to the same underlying entity. If LMs merely rely on surface-level statistics, they might select antecedents based solely on gender or number matching rather than a deeper "semantico-referential" understanding. This research investigates whether LM hidden representations encode the link between words and the entities they represent.

Methodology: Diagnostic Probing

To investigate internal representations, the authors employ probe tasks (diagnostic classifiers). By training a shallow neural network on the hidden layers of a "frozen" pre-trained LM, researchers can determine if specific information—in this case, coreference—is present in the model's state.

Two primary architectures were evaluated:

  1. AWD-LSTM: A standard recurrent architecture.
  2. TransformerXL: A state-of-the-art attention-based model.

The experiments focused on two scales: a Local Probe, identifying antecedents within a 60-token window, and a Global Probe, determining if pronouns across an entire document refer to the same entity regardless of form.

Experimental Setup

The study utilizes the OntoNotes 5.0 corpus, which contains extensive coreference chain annotations. To ensure robust evaluation of rare linguistic phenomena, the researchers reversed the standard data partitions, utilizing the larger original test set for probe training. The analysis specifically targeted narrative text to avoid the complexities inherent in dialogue.

Analysis of Referential Encoding

The study revealed several critical insights into how LMs handle entity tracking:

  • Beyond Morphosyntax: While models rely heavily on grammar, they also capture referential aspects. The authors suggest that because a single entity underlies multiple mentions, the surrounding coherent contexts allow the LM to "triangulate" the entity.
  • Recency vs. Attention: LSTMs exhibit a strong "recency bias," frequently defaulting to the most recent mention. In contrast, the attention mechanism in Transformers allows for more effective long-range entity tracking.
  • Graceful Degradation: The models do not rely on simple heuristics. When "distractors" (entities with matching gender/number) are introduced, performance decreases but does not collapse, indicating the use of broader contextual clues.
Figure 2: Model accuracy degrades gracefully as the proportion of gold mentions decreases, suggesting a reliance on more than just surface-level heuristics.

Comparative Performance

The results highlight a clear hierarchy in model performance. Both neural architectures outperformed a strong rule-based baseline (61.3%) that simply selects the nearest word of the same form.

Figure 4: Comparison of accuracy across models. The TransformerXL significantly outperforms the LSTM in supervised probing tasks.

Key performance metrics include:

  • Transformer Dominance: In local probing, TransformerXL achieved 75.9% accuracy, significantly higher than the AWD-LSTM's 64.8%.
  • Mention Identification: The Transformer was exceptionally proficient at identifying coreferent words, reaching 92.6% accuracy.
  • The Global Challenge: Performance dropped in global tasks. Without linear distance cues, models struggled to distinguish between different entities represented by the same pronoun (e.g., two different "she" mentions in a single story).

Conclusion and Future Directions

While LMs demonstrate "primitive referential abilities," they remain susceptible to confusion from morphosyntactic distractors. Future research will explore CNN-based architectures, the emergence of these abilities during training, and methods to explicitly foster referential tracking to achieve truer natural language understanding.

Summary updated

BibTeX
@inproceedings{sorodoc2020probing,
  author = {Ionut-Teodor Sorodoc and Kristina Gulordava and Gemma Boleda},
  title = {{Probing for Referential Information in Language Models}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2020},
  doi = {10.18653/v1/2020.acl-main.384},
}
Figure