Abstract
This paper presents the first study on using large-scale pre-trained language models for automated generation of an event-level temporal graph for a document. Despite the huge success of neural pre-training methods in NLP tasks, its potential for temporal reasoning over event graphs has not been sufficiently explored. Part of the reason is the difficulty in obtaining large training corpora with human-annotated events and temporal links. We address this challenge by using existing IE/NLP tools to automatically generate a large quantity (89,000) of system-produced document-graph pairs, and propose a novel formulation of the contextualized graph generation problem as a sequence-to-sequence mapping task. These strategies enable us to leverage and fine-tune pre-trained language models on the system-induced training data for the graph generation task. Our experiments show that our approach is highly effective in generating structurally and semantically valid graphs. Further, evaluation on a challenging hand-labeled, out-domain corpus shows that our method outperforms the closest existing method by a large margin on several metrics.
Introduction
Understanding the chronological order of events—who did what, and in what order—is a fundamental component of reading comprehension. In natural language processing (NLP), this is typically represented as a temporal graph, where nodes denote events and edges represent relations such as "before," "after," or "simultaneous."
Traditionally, constructing these graphs required complex pipelines involving part-of-speech tagging, entity recognition, coreference resolution, and relation classification. These pipelines are often brittle, as errors propagate through each stage. Moreover, the scarcity of human-annotated data (often limited to a few hundred documents) has hindered the application of modern transformer-based models to this task. This research explores how to bypass these pipelines by treating graph generation as an end-to-end sequence-to-sequence (seq2seq) task.
Methodology
Graph as Language
To bridge the gap between structured graphs and language models, the authors represent temporal graphs using the DOT language, a plain-text graph description format. This allows a document to be treated as input and its corresponding temporal structure as a target string for translation. By listing edges in the DOT string in the same order that events appear in the text, the model benefits from a "locality bias" that aids the attention mechanism.
Data Silver-Sourcing and Refinement
To overcome data scarcity, the authors utilized CAEVO, an existing temporal information extraction tool, to process 89,000 New York Times articles. This resulted in a "silver-standard" dataset of document-graph pairs (TG-Gen). The data was further refined by pruning noisy reporting verbs (e.g., "said") and augmenting simple verbs into more descriptive "event phrases." To manage complexity, community detection algorithms were applied to decompose large document graphs into smaller, contextually dense sub-graphs.
Model Fine-Tuning
The team fine-tuned GPT-2 using a masked loss function. This approach ensures the model is penalized only for errors in the generated DOT string, rather than the input text, focusing the learning process on the structural mapping task.
Experimental Results
Quantitative Performance
GPT-2 demonstrated a significant performance leap over traditional Seq2Seq LSTM baselines. On the TG-Gen test set, GPT-2 achieved a BLEU score of 62.53, compared to 21.23 for the baseline. The model also showed strong zero-shot generalization when tested on the human-labeled TimeBank-Dense (TB-Dense) dataset.
Key Insights
- The Teacher vs. Student Paradox: Remarkably, the "student" model (GPT-2) eventually outperformed the "teacher" tool (CAEVO) used to generate its training data. By training on a massive scale, the model learned to generalize beyond the noise and specific errors of the automated source, even "fixing" labels in about 10% of cases during human evaluation.
- Structural vs. Semantic Accuracy: The model achieved 94.6% parse accuracy for the generated DOT files, indicating that LLMs can readily learn the syntax of graph languages. However, precise semantic extraction of every temporal link remains a more complex challenge.
- Zero-Shot Capabilities: Without task-specific training for question answering, the model could generate plausible answers to temporal queries (e.g., "What happened before X?") in approximately 32% of cases when tested on the Torque dataset.
Conclusion
This study demonstrates that reformulating temporal graph construction as a contextualized generation task allows LLMs to effectively capture the "flow of time" in text. While the model generalizes well to out-of-domain data, performance drops are observed when moving from news articles to more diverse text types. Future research will explore larger models and methods to handle varying levels of temporal granularity.
Summary updated