Abstract
Aman Madaan, Yiming Yang. Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies.
2021.
Introduction
Temporal reasoning—the ability to understand the chronological order of events—is a fundamental component of reading comprehension. Traditionally, these relationships are modeled as temporal graphs, where nodes represent events and edges denote temporal links such as "before," "after," or "simultaneously."
Automatically constructing these graphs is challenging due to data scarcity and the fragility of multi-stage pipelines. Human-annotated datasets are often too small for deep learning, and traditional rule-based systems suffer from error propagation across sub-components like dependency parsers and taggers. This paper proposes a novel end-to-end approach that treats temporal graph generation as a sequence-to-sequence translation task.
Methodology: Graphs as Sequences
The core innovation lies in representing temporal graphs using the DOT language, a standard plain-text format. By converting a graph into a string, the authors can leverage large-scale pre-trained language models like GPT-2.
Graph-as-String Representation
Instead of specialized graph neural networks, the model is trained to map input text directly to a DOT string. This allows the model to utilize its pre-existing linguistic knowledge to maintain structural constraints and semantic coherence. The authors also introduced a "locality bias" by listing edges in the order events appear in the text, aligning the task with the left-to-right processing of Transformer models.
Silver-Labeling and Pruning
To overcome the lack of large-scale annotated data, the researchers generated a "silver" dataset (TG-Gen) by applying an existing tool, CAEVO, to 89,000 documents from the New York Times. They refined this data through pruning—removing non-essential verbs like "said" or "told"—and community detection to cluster related events, creating a cleaner training signal.
Results and Evaluation
The proposed approach demonstrates significant improvements over traditional baselines. Fine-tuned GPT-2 models outperformed standard Seq2Seq (LSTM) architectures by a wide margin in both structural accuracy and semantic relevance.
Key findings include:
- High Structural Integrity: The model produced valid, parsable DOT files in 94.6% of cases, demonstrating its ability to learn complex graph syntax.
- Superior Metrics: On the TG-Gen dataset, GPT-2 achieved a BLEU score of 62.53, nearly triple the performance of the Seq2Seq baseline (21.23).
- Generalization: Despite being trained on synthetic data, the model outperformed specialized pipelines on the human-labeled TB-Dense dataset, which served as a rigorous out-of-domain test.
- Zero-Shot Reasoning: Human evaluation on the Torque dataset showed that the model could plausibly answer temporal questions it wasn't explicitly trained for, suggesting an emergent understanding of temporal "common sense."
Conclusion
By reframing graph generation as a language modeling task, this work bypasses the limitations of traditional pipelines. The success of GPT-2 in this domain highlights the potential of large-scale pre-training for complex structural reasoning tasks. Future work may explore handling longer documents and varying levels of temporal granularity.
Summary updated