Be Consistent! Improving Procedural Text Comprehension using Label Consistency

Xinya Du, Bhavana Dalvi Mishra, Niket Tandon, Antoine Bosselut, Wen-tau Yih, Peter Clark, Claire Cardie

Introduction

Procedural text comprehension—the ability to track how entities change state over time—is a fundamental challenge in natural language processing. Whether reading a scientific explanation of photosynthesis or a cooking recipe, a model must determine if an entity was created, destroyed, moved, or remained unchanged at each step. This task is difficult due to the inherent complexity of multi-step processes and the scarcity of detailed, step-by-step annotations.

Researchers from Cornell and the Allen Institute for AI (AI2) have introduced LaCE (Label Consistency Explorer), a framework that leverages "group wisdom" to overcome these hurdles. By enforcing consistency across multiple descriptions of the same process, LaCE achieves state-of-the-art results and enables effective learning from unlabeled data.

The LaCE Framework: Summary Labels and Consistency

The core innovation of LaCE is the use of Summary Labels. Instead of treating each step in isolation, the model computes an aggregate label for each entity across the entire passage. For instance, if a text describes carbon dioxide being absorbed and then consumed to create sugar, its summary label reflects both its movement and its eventual destruction.

LaCE operates on the principle that while different texts may vary in granularity or wording, they should agree on the high-level outcome.

Consistency across multiple descriptions of the same process

As shown in the figure above, multiple independent descriptions of photosynthesis (e.g., from different textbooks) might use different sentences, but the resulting state changes for entities like CO2, Sugar, and Oxygen should remain consistent. This "consistency bias" acts as a powerful regularizer, helping the model resolve ambiguities in one text by looking at others.

Training with Consistency Loss

During training, LaCE processes paragraphs in batches grouped by topic. The model optimizes a hybrid loss function:

  1. Supervised Loss ($L_{sup}$): Penalizes incorrect step-by-step predictions on labeled data.
  2. Consistency Loss ($L_{con}$): Penalizes disagreements between the predicted summary labels of different paragraphs within the same topic group.
The LaCE training loop using consistency loss across batches

To ensure stable training, the authors employ an adaptive loss strategy similar to simulated annealing. Initially, the model focuses solely on supervised learning. Once it gains a basic understanding of the task, the consistency loss is gradually introduced to refine its logic and enforce global coherence.

Key Results and Insights

LaCE was evaluated on the ProPara benchmark, a dataset of scientific processes. The framework demonstrated significant improvements over existing methods:

  • State-of-the-Art Performance: LaCE achieved an F1 score of 56.6, surpassing the previous best model (ProStruct) while being 15x faster to train due to a simplified architecture that removes the need for complex external knowledge bases during decoding.
  • Semi-Supervised Learning: One of the most significant advantages of LaCE is its ability to utilize unlabeled data. By including 877 additional unlabeled paragraphs in the training batches, the model uses consistency to "self-label" through consensus. In low-data scenarios (using only 33% of labels), adding unlabeled data via LaCE boosted the F1 score from 47.3 to 49.9.
  • Ablation Studies: Removing the consistency loss resulted in a performance drop to 53.2, confirming that the cross-text consistency constraint is the primary driver of the model's success.

Conclusion

LaCE demonstrates that consistency is a powerful signal for understanding complex procedural texts. By looking beyond individual paragraphs and enforcing agreement across multiple sources, models can better navigate the ambiguities of natural language. This framework opens new doors for semi-supervised learning in structured prediction tasks where expert annotation is difficult to obtain.

Summary updated

Abstract

Our goal is procedural text comprehension, namely tracking how the properties of entities (e.g., their location) change with time given a procedural text (e.g., a paragraph about photosynthesis, a recipe). This task is challenging as the world is changing throughout the text, and despite recent advances, current systems still struggle with this task. Our approach is to leverage the fact that, for many procedural texts, multiple independent descriptions are readily available, and that predictions from them should be consistent (label consistency). We present a new learning framework that leverages label consistency during training, allowing consistency bias to be built into the model. Evaluation on a standard benchmark dataset for procedural text, ProPara (Dalvi et al., 2018), shows that our approach significantly improves prediction performance (F1) over prior state-of-the-art systems.
BibTeX
@inproceedings{du2019be,
  author = {Xinya Du and Bhavana Dalvi Mishra and Niket Tandon and Antoine Bosselut and Wen-tau Yih and Peter Clark and Claire Cardie},
  title = {{Be Consistent! Improving Procedural Text Comprehension using Label Consistency}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2019},
  doi = {10.18653/v1/n19-1244},
}
Figure