Effective Use of Transformer Networks for Entity Tracking

Aditya Gupta, Greg Durrett

Aditya Gupta, Greg Durrett. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). 2019.

Abstract

This paper explores how to make Transformer-based models (like GPT and BERT) better at "entity tracking"—the ability to follow an object through a series of changes in procedural text, such as a recipe or a scientific description.

Introduction

Procedural language presents a unique challenge for NLP. In a recipe or a lab protocol, entities (like "eggs" or "carbon dioxide") undergo transformations: they are moved, destroyed, or merged into new compositions (like "the mixture"). While Transformers excel at general language understanding, they often fail at this specific task. If you simply feed a paragraph into BERT and then ask about an "egg" at the end (a method called post-conditioning), the model often performs worse than a simple rule-based baseline.

The Tracking Task

The authors evaluate their approach on two primary benchmarks that represent different tracking challenges:

  • RECIPES: A binary classification task where the goal is to predict if a specific ingredient is present or affected in each step of a cooking recipe. This requires tracking intermediate compositions and global state without explicit entity mentions.
  • ProPara: A structured prediction task involving scientific processes (e.g., photosynthesis). It requires tracking whether an entity is created, moved, or destroyed, and specifically where and when these events happen.
Figure 1: Comparison of the RECIPES (binary classification) and ProPara (structured prediction) entity tracking tasks.

Methodology: Entity-First Conditioning

The authors identify a gap: standard self-attention isn't naturally "entity-aware." To address this, they compare two main ways of using Transformers for tracking:

  1. Post-conditioning (Baseline): The model encodes the entire text independently. Only at the very last layer is a representation of the target entity introduced to "read off" the state.
  2. Entity-Conditioned Models (Innovation): The authors propose restructuring the input sequence to put the entity "front and center." Specifically, they use an Entity-First approach where the target entity is prepended to the process tokens.

By placing the entity at the beginning of the sequence ([START] Target Entity [SEP] Step 1 [CLS] ...), every token in the subsequent process can "attend" to that entity through every single layer of the Transformer. This allows the model to build a representation that is inherently conditioned on the entity being tracked.

Figure 2: The Entity-First architecture, where the target entity is prepended to the input sequence to enable entity-aware self-attention.

Key Insights and Results

Performance Gains

  • State-of-the-Art Results: On the RECIPES dataset, the Entity-First GPT model achieved an F1 score of 81.96, a massive jump over the standard post-conditioned GPT (67.60).
  • ProPara Success: The models outperformed previous specialized architectures (like EntNet or NCET). Notably, they saw a significant boost in "Cat-2" accuracy (identifying when a state change happens), moving from roughly 47% to 52.5%.

The "Verb-Watcher" Strategy

Through gradient-based analysis, the authors found that the models aren't necessarily building a complex "mental map" of the physical process. Instead, they become highly sensitive to specific verbs. For example, if the model sees the word "dredge," it strongly associates that with the presence of "butter" or "flour," even if those entities weren't explicitly mentioned in that sentence.

The Compositionality Gap

Despite high scores, models still struggle with "hidden" entities. If "eggs" and "flour" are mixed into "batter," the model is much less likely to realize the eggs are still there when the text later says "pour the batter." This suggests that while the models are effective, they rely heavily on shallow lexical clues rather than deep reasoning about physical states.

Conclusion

Simply prepending the entity name to the input was the "aha moment" that unlocked state-of-the-art performance for Transformers in procedural text. However, the authors note that the current approach is computationally expensive, as the entire Transformer must be re-run for every entity being tracked. Future work should focus on more efficient ways to inject entity-awareness and better handling of intermediate compositions.

Summary updated

BibTeX
@inproceedings{gupta2019effective,
  author = {Aditya Gupta and Greg Durrett},
  title = {{Effective Use of Transformer Networks for Entity Tracking}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2019},
  url = {https://openreview.net/forum?id=ro2XavaLhhny},
}
Figure