Procedural Text Understanding via Scene-Wise Evolution

Jialong Tang, Hongyu Lin, Meng Liao, Yaojie Lu, Xianpei Han, Le Sun, Weijian Xie, Jin Xu

Abstract

Procedural text understanding requires machines to reason about entity states within dynamical narratives. Current procedural text understanding approaches are commonly entity-wise, which separately track each entity and independently predict different states of each entity. Such an entity-wise paradigm does not consider the interaction between entities and their states. In this paper, we propose a new scene-wise paradigm for procedural text understanding, which jointly tracks states of all entities in a scene-by-scene manner. Based on this paradigm, we propose Scene Graph Reasoner (SGR), which introduces a series of dynamically evolving scene graphs to jointly formulate the evolution of entities, states, and their associations throughout the narrative. In this way, the deep interactions between all entities and states can be jointly captured and simultaneously derived from scene graphs. Experiments show that SGR not only achieves the new state-of-the-art performance but also significantly accelerates the speed of reasoning.

Introduction

Procedural text understanding is a fundamental challenge in natural language processing, requiring models to track the state and location of entities as they change throughout a narrative. Whether following a scientific process like photosynthesis or a culinary recipe, a model must understand how actions transform the world state.

Traditionally, this has been approached through an entity-wise paradigm, where models track each entity (e.g., "water," "minerals," "sunlight") independently. However, this approach suffers from two critical limitations. First, it ignores entity interactions; if a text describes two ingredients being mixed, an entity-wise model may fail to synchronize their locations. Second, it is computationally inefficient, as the complexity grows linearly with the number of entities. To address these issues, we introduce the Scene Graph Reasoner (SGR), shifting the focus from individual actors to the evolution of the entire scene.

The Scene-Wise Paradigm

Instead of maintaining separate timelines for every entity, SGR models the world state at each timestep $t$ as a Scene Graph. In this graph, nodes represent concepts—including entities, locations, and external knowledge—while edges represent the relationships between them (e.g., "located in" or "part of"). By treating the narrative as a sequence of evolving graphs, the model can capture the global state of the environment simultaneously.

Architecture of SGR

The SGR framework consists of four primary components designed to handle graph evolution:

  1. Graph Structure Encoder: This module utilizes a Graph Attention Network (GAT) to process the current scene graph, summarizing the existing state into a global representation.
  2. Context Encoder: A BERT-based module that processes the natural language description of the current step, identifying new actions or events that will trigger state changes.
  3. Graph Structure Predictor: An autoregressive module that combines the current scene representation and the new context to predict the subsequent scene graph. It specifically predicts a Node Mask (to determine which entities exist) and Edge Indications (to determine entity-location relationships).
  4. State Reasoner: This final module compares the scene graphs at $t-1$ and $t$ to derive specific state changes, such as whether an entity was "Created," "Moved," or "Destroyed."

To enhance its reasoning capabilities, SGR incorporates external commonsense knowledge from ConceptNet. This allows the model to understand physical constraints—for instance, recognizing that a "root" is a component of a "plant," which logically dictates where water absorbed by the root might be located.

Experimental Evaluation

We evaluated SGR on two major benchmarks for procedural reasoning: ProPara, which focuses on scientific processes, and Recipes, which tracks ingredient locations in cooking instructions.

Performance and Efficiency

SGR established new state-of-the-art results across both datasets:

  • ProPara: SGR achieved a 72.2 F1 score at the document level, surpassing the previous best model, REAL (70.5 F1).
  • Recipes: The model reached an 58.4 F1 score, a significant improvement over previous baselines which typically ranged between 54 and 56 F1.

Beyond accuracy, SGR demonstrated a substantial increase in reasoning speed. By predicting the states of all entities in parallel for a given scene, it avoids the bottleneck of entity-wise processing. On an Nvidia TITAN RTX, SGR processed the ProPara test set in 17.99 seconds, compared to 51.31 seconds for NCET and 42.50 seconds for IEN, representing a 2.3x to 2.8x speedup.

Conclusion

The transition from entity-wise tracking to a scene-wise paradigm represents a significant step forward in procedural text understanding. By modeling the narrative as a series of evolving scene graphs, SGR effectively captures the complex interactions between entities while providing a more efficient computational framework. Future work will look to address exposure bias in autoregressive graph prediction and explore the integration of multi-modal or hierarchical relations to further refine the model's understanding of dynamic environments.

Summary updated

Abstract

Procedural text understanding requires machines to reason about entity states within the dynamical narratives. Current procedural text understanding approaches are commonly \textbf{entity-wise}, which separately track each entity and independently predict different states of each entity. Such an entity-wise paradigm does not consider the interaction between entities and their states. In this paper, we propose a new \textbf{scene-wise} paradigm for procedural text understanding, which jointly tracks states of all entities in a scene-by-scene manner. Based on this paradigm, we propose \textbf{S}cene \textbf{G}raph \textbf{R}easoner (\textbf{SGR}), which introduces a series of dynamically evolving scene graphs to jointly formulate the evolution of entities, states and their associations throughout the narrative. In this way, the deep interactions between all entities and states can be jointly captured and simultaneously derived from scene graphs. Experiments show that SGR not only achieves the new state-of-the-art performance but also significantly accelerates the speed of reasoning.
BibTeX
@inproceedings{tang2022procedural,
  author = {Jialong Tang and Hongyu Lin and Meng Liao and Yaojie Lu and Xianpei Han and Le Sun and Weijian Xie and Jin Xu},
  title = {{Procedural Text Understanding via Scene-Wise Evolution}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2022},
  url = {https://openreview.net/forum?id=5GplcUjOZ4a},
}

Loading related content...

Figure