Conditional Generation of Temporally-ordered Event Sequences

Shih-Ting Lin, Nathanael Chambers, Greg Durrett

Understanding the "common sense" flow of time is a hallmark of human intelligence. We know that you must buy a gift before you can open it, and that someone usually gives it to you in between. While AI has made massive strides in language modeling, capturing these temporal relationships—the "narrative schemas" of our lives—remains a significant challenge.

In their paper, "Conditional Generation of Temporally-ordered Event Sequences," researchers from the University of Texas at Austin and the U.S. Naval Academy present TemporalBART, a model designed to master the chronological logic of events.

TL;DR

TemporalBART treats temporal reasoning as a sequence generation problem rather than a simple classification task. By training a BART-based model as a denoising autoencoder on shuffled and incomplete event sequences, the authors created a system that can both sort scrambled events into their correct chronological order and "infill" missing steps in a scenario with high coherence.

What Problem Does This Paper Address?

Most existing work on "narrative schemas" focuses on event co-occurrence—knowing that "buying," "giving," and "opening" often happen in the same story. However, these models rarely capture the strict temporal constraints between those events.

Traditional approaches to temporal ordering usually rely on pairwise classification (deciding if event A happened before B) or local discourse cues (words like "then" or "after" found in a specific text). The problem is that these models often fail to learn generalizable temporal knowledge. If you strip away the connector words, the models get lost. The authors sought a single, flexible architecture that understands the underlying "script" of events, independent of specific document context.

Methods

The researchers adapted BART, a powerful transformer-based encoder-decoder model, to act as a denoising autoencoder.

The training process involves a clever "corruption" strategy (illustrated in Figure 2):

  1. Event Shuffling: A gold-standard chronological sequence of events is randomly scrambled.
  2. Event Deletion: Some events are removed entirely.
  3. Reconstruction: The model is tasked with outputting the full, correctly ordered sequence.
Figure 2

To help the model distinguish between events it should "copy" from the input and events it needs to "invent" (infill), they introduced TemporalBART-indexed. This version uses special tags (like [E1], [E2]) to track specific input events, allowing the model to focus its generative power on the temporal structure rather than just re-typing the text.

The model architecture for TemporalBART-indexed

Datasets

  • Training: The model was trained on EventsNarratives, a large-scale corpus of 200,000 sequences extracted from news, books, and blogs. The authors used a heuristic assuming that in narrative text, the order of mention (discourse order) roughly reflects the order of occurrence.
  • Evaluation (Ordering): To test the model's ability to sort events, they used CaTeRS (short stories with causal/temporal annotations) and MCTaco (a temporal common sense QA dataset).
  • Evaluation (Infilling): Human raters evaluated the model's ability to generate plausible missing events using sequences from the CaTeRS dataset.

Key Insights and Ideas

  • Ordering as Generation: By treating ordering as a sequence generation task, the model can capture global dependencies across a whole chain of events, which is much more powerful than looking at pairs of events in isolation.
  • The Power of Deletion: The "Event Deletion" step in training is the "aha moment" for infilling. It forces the model to learn what should be there, effectively teaching it the "missing links" in common human activities.
  • Out-of-Context is Better: By training on events stripped of their original sentences and discourse markers, the model is forced to learn the actual semantics of the events (e.g., "opening" must follow "receiving") rather than relying on linguistic shortcuts.

Key Results

TemporalBART significantly outperformed state-of-the-art baselines across the board:

  • Superior Ordering: On the CaTeRS dataset, TemporalBART-indexed achieved a pairwise accuracy of 79.7%, crushing a BERT-based pointer network (54.1%) and a pairwise SSVM model (65.7%).
  • Better Infilling: In human evaluations, TemporalBART's generated events were rated significantly higher for "temporality" (1.10 on a 0-2 scale) compared to GPT-2 (0.57). While GPT-2 can generate coherent text, it often fails to place that text in the correct chronological slot.
  • Zero-Shot Success: The model performed remarkably well on out-of-domain datasets without any task-specific fine-tuning, suggesting it truly learned general temporal principles.
Examples of events generated by different models

Limitations and Future Work

The authors acknowledge a few hurdles:

  • The Discourse Assumption: Training relies on the assumption that writers list events in order. While generally true, flashbacks or complex narrative structures in the training data could introduce "noise" into the model's temporal understanding.
  • Surface Realization: Because the model works with predicate-argument structures, the final output isn't always a perfectly fluid, grammatical sentence, though it is semantically correct.
  • Complex Scenarios: While the model excels at linear sequences, future work could explore more complex "event graphs" where multiple timelines branch or merge.

Summary updated

Abstract

Models of narrative schema knowledge have proven useful for a range of event-related tasks, but they typically do not capture the temporal relationships between events. We propose a single model that addresses both temporal ordering, sorting given events into the order they occurred, and event infilling, predicting new events which fit into an existing temporally-ordered sequence. We use a BART-based conditional generation model that can capture both temporality and common event co-occurrence, meaning it can be flexibly applied to different tasks in this space. Our model is trained as a denoising autoencoder: we take temporally-ordered event sequences, shuffle them, delete some events, and then attempt to recover the original event sequence. This task teaches the model to make inferences given incomplete knowledge about the events in an underlying scenario. On the temporal ordering task, we show that our model is able to unscramble event sequences from existing datasets without access to explicitly labeled temporal training data, outperforming both a BERT-based pairwise model and a BERT-based pointer network. On event infilling, human evaluation shows that our model is able to generate events that fit better temporally into the input events when compared to GPT-2 story completion models.
BibTeX
@inproceedings{lin2021conditional,
  author = {Shih-Ting Lin and Nathanael Chambers and Greg Durrett},
  title = {{Conditional Generation of Temporally-ordered Event Sequences}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
  url = {https://openreview.net/forum?id=GYGhYgXgUs2},
}

Loading related content...

Figure