Attending to Entities for Better Text Understanding

Pengxiang Cheng, Katrin ErkAAAI 2020 · DOI

Introduction

While large-scale pre-trained language models like GPT-2 and BERT have revolutionized natural language processing, they often rely on surface-level statistical cues rather than deep linguistic reasoning. On tasks requiring long-distance dependency tracking—specifically, understanding who or what a pronoun refers to across several sentences—there remains a significant gap between machine and human performance.

This research focuses on the LAMBADA task, which requires predicting the last word of a narrative passage. These passages are specifically designed to be easy for humans with full context but impossible to solve using only the final sentence. To succeed, a model must track "entities" (people, places, things) across a discourse. The authors argue that instead of just increasing parameter counts, explicit linguistic knowledge—specifically coreference information—should be injected into the model's attention mechanism.

Methods

The researchers built upon the BiDAF (Bidirectional Attention Flow) architecture, a staple in reading comprehension, and enhanced it with Transformer-style Multi-Head Self-Attention (MHSA) layers. They explored different architectural placements for these layers, specifically comparing "Early" (contextual encoding stage) and "Late" (post-bidirectional attention) variants.

Architectural variants of the BiDAF model with added self-attention layers.

The core innovation is Supervised Self-Attention. Rather than letting the model learn attention weights entirely on its own, an auxiliary loss function "guides" specific attention heads to focus on linguistically relevant tokens. Three types of supervision were tested:

  1. DEPPARSE (Syntax): Encourages tokens to attend to their syntactic heads.
  2. COREFALL (Coreference): Encourages tokens to attend to all other mentions of the same entity.
  3. NARRATIVE: A hybrid approach where entity mentions attend to predicates associated with their coreferent mentions, facilitating event tracking.
Visualization of attention matrices for DEPPARSE, COREFALL, and NARRATIVE supervision.

Crucially, "silver" labels generated by off-the-shelf NLP toolkits (Stanford CoreNLP) were used for supervision, demonstrating that even noisy automated signals provide a powerful inductive bias.

Key Insights and Results

The most striking finding is the efficiency of this approach. The best-performing model uses roughly 2.6 million tunable parameters, yet it achieved an average accuracy of 63.71% on LAMBADA, outperforming the 1.5 billion parameter GPT-2 model (63.24%).

Analysis confirmed that coreference supervision provided the most significant gains, particularly on instances involving pronouns and named entities. This supports the hypothesis that entity tracking is a primary bottleneck in narrative understanding. Furthermore, the "Early" architecture placement proved more effective, suggesting that entity tracking should be fundamental to sentence encoding rather than a post-processing step.

Conclusion

By explicitly teaching a small model to track entities through coreference supervision, researchers from UT Austin demonstrated that targeted linguistic inductive biases can outperform massive scale. Future work includes integrating this supervision into the pre-training phase of larger models and exploring other semantic signals like Semantic Role Labeling (SRL).

Summary updated

BibTeX
@inproceedings{cheng2020attending,
  author = {Pengxiang Cheng and Katrin Erk},
  title = {{Attending to Entities for Better Text Understanding}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2020},
  doi = {10.1609/aaai.v34i05.6254},
}
Figure