Abstract
Procedural text understanding requires machines to reason about entity states within the dynamical narratives. Current procedural text understanding approaches are commonly entity-wise, which separately track each entity and independently predict different states of each entity. Such an entity-wise paradigm does not consider the interaction between entities and their states. In this paper, we propose a new scene-wise paradigm for procedural text understanding, which jointly tracks states of all entities in a scene-by-scene manner. Based on this paradigm, we propose Scene Graph Reasoner (SGR), which introduces a series of dynamically evolving scene graphs to jointly formulate the evolution of entities, states and their associations throughout the narrative. In this way, the deep interactions between all entities and states can be jointly captured and simultaneously derived from scene graphs. Experiments show that SGR not only achieves the new state-of-the-art performance but also significantly accelerates the speed of reasoning.
Introduction: The Challenge of Procedural Text
Procedural text understanding involves tracking how entities change state—such as being created, moved, or destroyed—and location throughout a narrative. Common examples include scientific descriptions of natural processes (e.g., photosynthesis) or culinary recipes.
Traditionally, models have followed an entity-wise paradigm. In this approach, the model tracks each entity independently: it follows "water" through the entire text, then restarts to track "minerals," and so on. This method has two significant drawbacks:
- Ignoring Interactions: Entities in procedural texts often interact. For instance, if water and minerals combine into a mixture, their states become highly correlated. Entity-wise models struggle to capture these dependencies.
- Inefficiency: Processing each entity separately is computationally expensive. If a paragraph contains ten entities, the model must essentially process the narrative ten times.
Proposed Method: Scene Graph Reasoner (SGR)
To address these limitations, we introduce a scene-wise paradigm where the state of the entire "world" is modeled as a graph that evolves sentence-by-sentence. Our implementation, the Scene Graph Reasoner (SGR), models the evolution of entities, states, and their associations jointly.
The SGR architecture consists of four primary components:
- Graph Structure Encoder: Utilizes a Graph Attention Network (GAT) to summarize the current "scene." Nodes represent entities, locations, or common-sense concepts, while edges represent their relationships.
- Context Encoder: Employs a pre-trained BERT model to extract event information from the current sentence.
- Graph Structure Predictor: An autoregressive module that combines the scene summary and event info to predict the next scene graph (specifically, node existence and edge connectivity).
- State Reasoner: Determines specific actions (e.g., "Moved") by comparing adjacent scene graphs.
Crucially, SGR incorporates external common-sense knowledge from ConceptNet (e.g., knowing a "root" is part of a "plant") to better understand the physical constraints of the environment.
Evaluation and Results
We evaluated SGR on two primary benchmarks: ProPara (scientific processes) and Recipes (cooking instructions).
Performance and Efficiency
- State-of-the-Art Accuracy: SGR achieved a 72.2 F1 score on the ProPara document-level task, outperforming the previous best model, REAL (70.5 F1). On the Recipes dataset, SGR reached an F1 of 58.4, compared to 54.1 for the previous state-of-the-art.
- Massive Speedup: By predicting the states of all entities simultaneously in one pass per sentence, SGR eliminates redundant computation. On an Nvidia TITAN RTX, it processed ProPara in 17.99 seconds, whereas previous models like NCET and IEN took 51.31 and 42.50 seconds, respectively—a 3-4x increase in reasoning speed.
Key Insights
- Joint Reasoning: Modeling the entire scene at once allows the model to "understand" that if one entity moves, related entities are likely to interact or follow, mirroring human intuition.
- Evolutionary Modeling: Treating state tracking as dynamic graph evolution rather than simple classification allows the model to maintain a structurally consistent "memory" of the world state.
- Knowledge Synergy: Linguistic knowledge from BERT and relational knowledge from ConceptNet are complementary. BERT handles textual nuance, while ConceptNet provides physical world constraints.
Limitations and Future Work
While SGR shows significant improvements, it faces exposure bias due to its autoregressive nature; errors in early scene predictions can propagate through the narrative. Future work may involve pre-training graph-structured language models to maintain label consistency across complex, evolving topics. Additionally, the training process requires transforming existing datasets into gold-standard scene graphs, which increases preprocessing complexity.
Summary updated