Conditional Generation of Temporally-ordered Event Sequences

Shih-Ting Lin, Nathanael Chambers, Greg Durrett

Abstract

Models of narrative schema knowledge have proven useful for a range of event-related tasks, but they typically do not capture the temporal relationships between events. We propose a single model that addresses both temporal ordering, sorting given events into the order they occurred, and event infilling, predicting new events which fit into an existing temporally-ordered sequence. We use a BART-based conditional generation model that can capture both temporality and common event co-occurrence, meaning it can be flexibly applied to different tasks in this space. Our model is trained as a denoising autoencoder: we take temporally-ordered event sequences, shuffle them, delete some events, and then attempt to recover the original event sequence. This task teaches the model to make inferences given incomplete knowledge about the events in an underlying scenario. On the temporal ordering task, we show that our model is able to unscramble event sequences from existing datasets without access to explicitly labeled temporal training data, outperforming both a BERT-based pairwise model and a BERT-based pointer network. On event infilling, human evaluation shows that our model is able to generate events that fit better temporally into the input events when compared to GPT-2 story completion models.

Introduction

Understanding the "common sense" flow of time is a hallmark of human intelligence. We intuitively know that a gift must be purchased before it can be opened, and that the act of giving typically occurs in between. While modern AI has made significant strides in language modeling, capturing these temporal relationships—the "narrative schemas" of daily life—remains a challenge. Most existing work focuses on event co-occurrence rather than strict temporal constraints. Traditional approaches often rely on pairwise classification or local discourse cues (e.g., "then", "after"), which may not generalize well when such markers are absent. This research presents TemporalBART, a model designed to master the chronological logic of events by treating temporal reasoning as a sequence generation problem.

Methodology

The researchers adapted BART, a transformer-based encoder-decoder model, to act as a denoising autoencoder. The core idea is to train the model to reconstruct a coherent, chronological narrative from a corrupted input.

Training via Corruption

The training process involves a specific corruption strategy applied to gold-standard chronological sequences, as shown in the diagram below:

  1. Event Shuffling: The original sequence is randomly scrambled to remove temporal order.
  2. Event Deletion: A subset of events is removed entirely to simulate incomplete information.
  3. Reconstruction: The model is tasked with outputting the full, correctly ordered sequence, including the missing events.
The training process of TemporalBART as a denoising autoencoder involving event shuffling and deletion.

TemporalBART-indexed

To help the model distinguish between events it should "copy" from the input and events it needs to "invent" (infill), the authors introduced TemporalBART-indexed. This version uses special index tags (e.g., [E1], [E2]) to track specific input events. This allows the model to focus its generative capacity on the temporal structure and the content of missing events rather than simple text reproduction.

The model architecture for TemporalBART-indexed, showing how input events are tagged and reconstructed or infilled.

Experiments and Datasets

The model was trained on EventsNarratives, a large-scale corpus of 200,000 sequences extracted from news, books, and blogs, under the assumption that discourse order in narrative text generally reflects temporal order. Evaluation was conducted on two primary tasks:

  • Temporal Ordering: Tested using the CaTeRS (short stories with causal/temporal annotations) and MCTaco datasets.
  • Event Infilling: Evaluated by human raters who judged the plausibility and temporal fit of generated missing events in CaTeRS sequences.

Key Results

TemporalBART significantly outperformed state-of-the-art baselines in both tasks:

  • Superior Ordering: On the CaTeRS dataset, TemporalBART-indexed achieved a pairwise accuracy of 79.7%, surpassing a BERT-based pointer network (54.1%) and a pairwise SSVM model (65.7%).
  • Effective Infilling: In human evaluations, TemporalBART's generated events were rated significantly higher for "temporality" (1.10 on a 0-2 scale) compared to a GPT-2 story completion model (0.57). While GPT-2 generates coherent text, it often fails to place it in the correct chronological context.
  • Zero-Shot Generalization: The model performed well on out-of-domain datasets without task-specific fine-tuning, suggesting it learned fundamental temporal principles rather than dataset-specific biases.

Conclusion

By treating temporal ordering as a generation task and utilizing a denoising autoencoder objective, TemporalBART captures global dependencies across event chains more effectively than local pairwise models. Training on events stripped of discourse markers forces the model to learn the underlying semantics of event sequences. Future work may extend this approach to more complex event graphs involving branching or merging timelines.

Summary updated

Abstract

Models of narrative schema knowledge have proven useful for a range of event-related tasks, but they typically do not capture the temporal relationships between events. We propose a single model that addresses both temporal ordering, sorting given events into the order they occurred, and event infilling, predicting new events which fit into an existing temporally-ordered sequence. We use a BART-based conditional generation model that can capture both temporality and common event co-occurrence, meaning it can be flexibly applied to different tasks in this space. Our model is trained as a denoising autoencoder: we take temporally-ordered event sequences, shuffle them, delete some events, and then attempt to recover the original event sequence. This task teaches the model to make inferences given incomplete knowledge about the events in an underlying scenario. On the temporal ordering task, we show that our model is able to unscramble event sequences from existing datasets without access to explicitly labeled temporal training data, outperforming both a BERT-based pairwise model and a BERT-based pointer network. On event infilling, human evaluation shows that our model is able to generate events that fit better temporally into the input events when compared to GPT-2 story completion models.
BibTeX
@inproceedings{lin2021conditional,
  author = {Shih-Ting Lin and Nathanael Chambers and Greg Durrett},
  title = {{Conditional Generation of Temporally-ordered Event Sequences}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
  url = {https://openreview.net/forum?id=GYGhYgXgUs2},
}
Figure