Neural Language Modeling for Contextualized Temporal Graph Generation

Aman Madaan, Yiming Yang

Abstract

Aman Madaan, Yiming Yang. Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies.

2021.

Introduction

Temporal reasoning—the ability to understand the chronological order of events—is a fundamental component of reading comprehension. Traditionally, these relationships are modeled as temporal graphs, where nodes represent events and edges denote temporal links such as "before," "after," or "simultaneously."

Example of a temporal graph extracted from text, showing events and their temporal relationships.

Automatically constructing these graphs is challenging due to data scarcity and the fragility of multi-stage pipelines. Human-annotated datasets are often too small for deep learning, and traditional rule-based systems suffer from error propagation across sub-components like dependency parsers and taggers. This paper proposes a novel end-to-end approach that treats temporal graph generation as a sequence-to-sequence translation task.

Methodology: Graphs as Sequences

The core innovation lies in representing temporal graphs using the DOT language, a standard plain-text format. By converting a graph into a string, the authors can leverage large-scale pre-trained language models like GPT-2.

Graph-as-String Representation

Instead of specialized graph neural networks, the model is trained to map input text directly to a DOT string. This allows the model to utilize its pre-existing linguistic knowledge to maintain structural constraints and semantic coherence. The authors also introduced a "locality bias" by listing edges in the order events appear in the text, aligning the task with the left-to-right processing of Transformer models.

Silver-Labeling and Pruning

To overcome the lack of large-scale annotated data, the researchers generated a "silver" dataset (TG-Gen) by applying an existing tool, CAEVO, to 89,000 documents from the New York Times. They refined this data through pruning—removing non-essential verbs like "said" or "told"—and community detection to cluster related events, creating a cleaner training signal.

Results and Evaluation

The proposed approach demonstrates significant improvements over traditional baselines. Fine-tuned GPT-2 models outperformed standard Seq2Seq (LSTM) architectures by a wide margin in both structural accuracy and semantic relevance.

Performance comparison of GPT-2 against Seq2Seq baselines on TG-Gen and TB-Dense datasets.

Key findings include:

  • High Structural Integrity: The model produced valid, parsable DOT files in 94.6% of cases, demonstrating its ability to learn complex graph syntax.
  • Superior Metrics: On the TG-Gen dataset, GPT-2 achieved a BLEU score of 62.53, nearly triple the performance of the Seq2Seq baseline (21.23).
  • Generalization: Despite being trained on synthetic data, the model outperformed specialized pipelines on the human-labeled TB-Dense dataset, which served as a rigorous out-of-domain test.
  • Zero-Shot Reasoning: Human evaluation on the Torque dataset showed that the model could plausibly answer temporal questions it wasn't explicitly trained for, suggesting an emergent understanding of temporal "common sense."

Conclusion

By reframing graph generation as a language modeling task, this work bypasses the limitations of traditional pipelines. The success of GPT-2 in this domain highlights the potential of large-scale pre-training for complex structural reasoning tasks. Future work may explore handling longer documents and varying levels of temporal granularity.

Summary updated

BibTeX
@inproceedings{madaan2021neural,
  author = {Aman Madaan and Yiming Yang},
  title = {{Neural Language Modeling for Contextualized Temporal Graph Generation}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2021},
  url = {https://openreview.net/forum?id=lfD2bYvs6m3},
}
Figure