Effective Use of Transformer Networks for Entity Tracking

Aditya Gupta, Greg Durrett

Abstract

Tracking entities in procedural language requires understanding the transformations arising from actions on entities as well as those entities' interactions. While self-attention-based pre-trained language encoders like GPT and BERT have been successfully applied across a range of natural language understanding tasks, their ability to handle the nuances of procedural texts is still untested. In this paper, we explore the use of pre-trained transformer networks for entity tracking tasks in procedural text. First, we test standard lightweight approaches for prediction with pre-trained transformers, and find that these approaches underperform even simple baselines. We show that much stronger results can be attained by restructuring the input to guide the transformer model to focus on a particular entity. Second, we assess the degree to which transformer networks capture the process dynamics, investigating such factors as merged entities and oblique entity references. On two different tasks, ingredient detection in recipes and QA over scientific processes, we achieve state-of-the-art results, but our models still largely attend to shallow context clues and do not form complex representations of intermediate entity or process state.

Introduction

Tracking entities in procedural language—such as recipes or scientific descriptions—is a significant challenge for AI. Unlike standard narrative text, procedural text involves entities that undergo constant transformation: ingredients are mixed into batters, chemicals change state, and objects move between locations. Standard pre-trained Transformers, while excellent at syntax and local context, often struggle to maintain a "global" state of an entity throughout a sequence of actions.

Datasets and Tasks

The study evaluates models on two primary benchmarks:

  1. RECIPES: A binary classification task for ingredient detection. The goal is to determine if a specific ingredient (e.g., sugar, eggs) is present at any given step.
  2. ProPara: A structured prediction task for state changes in scientific processes. It tracks whether entities are created (C), moved (M), or destroyed (D) across steps.
Comparison of entity tracking tasks in Recipes and ProPara datasets.

Methodology: From Post-Conditioning to Entity-Centric Input

The researchers compared two primary architectural approaches for using Transformers in entity tracking:

Post-conditioning (The Standard Approach)

In this setup, the model encodes the entire text first. A separate query about a specific entity is then applied to the output. This means the Transformer does not "know" which entity it is tracking while processing the procedural steps.

Entity-Centric Input (The Proposed Approach)

To overcome the limitations of post-conditioning, the authors proposed restructuring the input to be entity-centric. By placing the target entity at the beginning of the input sequence (e.g., [START] Target Entity [SEP] Step 1 [CLS] ...), the Transformer’s self-attention mechanism is forced to relate every subsequent word back to that specific entity from the outset.

The entity-centric input architecture, where the target entity is provided before the process tokens to condition the contextual embeddings.

Key Results and Insights

The shift to an entity-centric input yielded significant performance gains:

  • RECIPES: The proposed ET_GPT (Entity-centric GPT) model achieved an F1 score of 81.96, a substantial improvement over the post-conditioning model (68.04) and rule-based baselines (74.60).
  • ProPara: The models achieved state-of-the-art results, reaching a Macro-Avg of 63.07, outperforming previous specialized architectures like NCET.

Despite these high scores, analysis revealed that the models often rely on "shallow" tracking. They become adept at identifying verbs like "add" or "mix" as signals for state changes but struggle with "aliasing"—where individual ingredients (like eggs and flour) become a single "mixture." When entities are part of intermediate compositions that aren't explicitly named, model accuracy drops significantly.

Conclusion

This research demonstrates that Transformer models can be highly effective for entity tracking in procedural text, provided the input is structured to guide the model's attention toward specific entities. While these models achieve state-of-the-art performance through sophisticated "verb-tracking," they still lack a deep mental representation of composed states. Future work should focus on developing models that can better represent and track entities through complex interactions and intermediate compositions.

Summary updated

Abstract

Tracking entities in procedural language requires understanding the transformations arising from actions on entities as well as those entities' interactions. While self-attention-based pre-trained language encoders like GPT and BERT have been successfully applied across a range of natural language understanding tasks, their ability to handle the nuances of procedural texts is still untested. In this paper, we explore the use of pre-trained transformer networks for entity tracking tasks in procedural text. First, we test standard lightweight approaches for prediction with pre-trained transformers, and find that these approaches underperform even simple baselines. We show that much stronger results can be attained by restructuring the input to guide the transformer model to focus on a particular entity. Second, we assess the degree to which transformer networks capture the process dynamics, investigating such factors as merged entities and oblique entity references. On two different tasks, ingredient detection in recipes and QA over scientific processes, we achieve state-of-the-art results, but our models still largely attend to shallow context clues and do not form complex representations of intermediate entity or process state.
BibTeX
@inproceedings{gupta2019effective,
  author = {Aditya Gupta and Greg Durrett},
  title = {{Effective Use of Transformer Networks for Entity Tracking}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2019},
  url = {https://openreview.net/forum?id=ro2XavaLhhny},
}
Figure