TRAM: Benchmarking Temporal Reasoning for Large Language Models

Yuqing Wang, Yun ZhaoSubmission to ICLR 2024 · Rejected

Executive Summary

TRAM is a comprehensive new benchmark designed to evaluate how well Large Language Models (LLMs) understand time. Spanning over 526,000 questions across ten diverse tasks, it reveals that while models like GPT-4 demonstrate impressive capabilities, they still lag more than 10% behind human experts—particularly when dealing with implicit temporal cues and complex causality.

The Challenge of Temporal Reasoning

Reasoning about time—knowing that a sunset is shorter than a movie, or that "before long" doesn't mean "immediately"—is second nature to humans but a minefield for artificial intelligence. While LLMs have made giant leaps in general language tasks, their Temporal Reasoning (TeR) capabilities remain inconsistent.

The authors identify a major gap in the field: previous research has been highly fragmented. Existing benchmarks often focus on narrow slices of time, such as simple event ordering or duration, without a unified framework. Without a standardized "stress test" that covers everything from basic clock arithmetic to the nuances of temporal storytelling, researchers cannot accurately measure or improve how AI perceives the flow of events.

Methodology

To address this, the researchers introduced TRAM (Temporal Reasoning for large lAnguage Models), a comprehensive suite of 10 tasks and 38 subtasks. To ensure clean, objective evaluation, all tasks are formatted as multiple-choice questions (MCQs), avoiding the grading ambiguity often associated with open-ended text generation.

The study evaluated a robust lineup of models:

  • Proprietary LLMs: GPT-4 Turbo, GPT-3.5 Turbo, and Gemini Pro.
  • Open-Source LLMs: Llama-2-70b-chat.
  • Baselines: BERT and RoBERTa models, plus a domain-specific model called RST.

Evaluations utilized two main strategies: Standard Prompting (SP) and Chain-of-Thought (CoT), across both zero-shot and 5-shot scenarios. The team also performed a deep-dive manual error analysis to categorize exactly why models fail.

The TRAM Benchmark

TRAM acts as a "benchmark of benchmarks," aggregating and augmenting data from several famous sources while adding original, human-curated templates.

Examples of TRAM multiple-choice questions across various temporal reasoning tasks.

The benchmark is divided into three main categories:

  1. Foundational Tasks: Ordering, Frequency, Duration, and Typical Time (sourced from MCTACO, Wikipedia, and SQuAD).
  2. Interpretation & Computation: Ambiguity Resolution and Arithmetic (using original templates for tasks like 24-hour time adjustments and time zone conversions).
  3. Advanced Reasoning: Temporal NLI, Relation, Causality, and Storytelling (sourced from TempEval-3, SNLI/MNLI, COPA, and ROCStories).

In total, the benchmark comprises 526,668 problems, making it one of the most robust datasets for temporal reasoning to date.

Key Insights & Error Analysis

  • Size Isn't Everything: In a surprising twist, RoBERTa-large (a much smaller, older architecture) achieved a higher average performance (65.9%) than the massive Llama-2-70b (65.6%) when given minimal supervision. This suggests that bidirectional architectures might be inherently better at picking up token-level temporal cues.
  • The "Human Gap" Persists: Even the best-performing model, GPT-4, remains roughly 12.9% behind human experts. Humans are particularly superior in "Foundational" tasks where common sense is required.
  • CoT is a Time-Saver: Chain-of-Thought prompting (asking the model to "think step-by-step") consistently boosted performance, especially in complex tasks like arithmetic and storytelling.
Error type distributions across Foundational, Interpretation/Computation, and Advanced temporal tasks.
  • The Nature of Error: Models fail in predictable ways depending on the task complexity. In foundational tasks, they suffer heavily from Assumption Bias (over-relying on training patterns). In interpretation and computation tasks, Calculation Slips dominate. Finally, in advanced tasks, models struggle most with Implicit Oversights, missing subtle clues that change the entire temporal context of a narrative.

Performance Highlights

  • Top Performer: GPT-4 (5-shot CoT) led the pack with an average accuracy of 84.4%.
  • Human Benchmark: Human experts scored 95.2%, setting a high bar for future AI development.
  • Arithmetic Variance: This was the most volatile category. While GPT-4 hit 91.2%, Llama-2 struggled significantly, hovering around 57.8%.
  • Temporal Relation: BERT-style models excelled here (up to 91.5% accuracy), likely due to their bidirectional nature being well-suited for identifying specific links between events in a sentence.

Limitations & Future Directions

The authors acknowledge that the MCQ format might allow models to "guess" their way to a better score. Furthermore, TRAM is purely text-based, whereas real-world temporal reasoning often involves multi-modal cues like video or audio.

Future directions for this research include:

  1. Generative Tasks: Expanding the benchmark to include short-answer and summarization tasks.
  2. Multi-modal TRAM: Incorporating images and video to test "visual" temporal reasoning.
  3. Expert Models: Fine-tuning newer open-source models (e.g., Llama-3 or Phi-3) specifically on TRAM to create dedicated "temporal expert" models.

Summary updated

Abstract

Reasoning about time is essential for understanding the nuances of events described in natural language. Previous research on this topic has been limited in scope, characterized by a lack of standardized benchmarks that would allow for consistent evaluations across different studies. In this paper, we introduce TRAM, a temporal reasoning benchmark composed of ten datasets, encompassing various temporal aspects of events such as order, arithmetic, frequency, and duration, designed to facilitate a comprehensive evaluation of the temporal reasoning capabilities of large language models (LLMs). We conduct an extensive evaluation using popular LLMs, such as GPT-4 and Llama2, in both zero-shot and few-shot learning scenarios. Additionally, we employ BERT-based models to establish the baseline evaluations. Our findings indicate that these models still trail human performance in temporal reasoning tasks. It is our aspiration that TRAM will spur further progress in enhancing the temporal reasoning abilities of LLMs.
BibTeX
@misc{wang2023tram,
  author = {Yuqing Wang and Yun Zhao},
  title = {{TRAM: Benchmarking Temporal Reasoning for Large Language Models}},
  year = {2023},
  eprint = {2310.00835},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2310.00835},
}
Figure