Living in the Moment: Can Large Language Models Grasp Co-Temporal Reasoning?

Zhaochen Su, Juntao Li, Jun Zhang, Tong Zhu, Xiaoye Qu, Pan Zhou, Yan Bowen, Yu Cheng, Min Zhang

Abstract

Temporal reasoning is fundamental for large language models (LLMs) to comprehend the world. Current temporal reasoning datasets are limited to questions about single or isolated events, falling short in mirroring the realistic temporal characteristics involving concurrent nature and intricate temporal interconnections. In this paper, we introduce CoTempQA, a comprehensive co-temporal Question Answering (QA) benchmark containing four co-temporal scenarios (Equal, Overlap, During, Mix) with 4,748 samples for evaluating the co-temporal comprehension and reasoning abilities of LLMs. Our extensive experiments reveal a significant gap between the performance of current LLMs and human-level reasoning on CoTempQA tasks. Even when enhanced with Chain of Thought (CoT) methodologies, models consistently struggle with our task. In our preliminary exploration, we discovered that mathematical reasoning plays a significant role in handling co-temporal events and proposed a strategy to boost LLMs' co-temporal reasoning from a mathematical perspective. We hope that our CoTempQA datasets will encourage further advancements in improving the co-temporal reasoning capabilities of LLMs.

Introduction

Temporal reasoning is a cornerstone of human intelligence, allowing us to navigate complex histories, follow news cycles, and understand multi-agent environments. While Large Language Models (LLMs) have demonstrated an ability to answer simple chronological questions (e.g., "What happened after X?"), they often falter when faced with concurrent events—the "meanwhile" of human existence. Real-world scenarios are rarely linear; they involve overlapping timelines and simultaneous occurrences. For instance, understanding who else was working at a specific company while a particular CEO was in office requires more than just identifying a date; it requires reasoning across intersecting time intervals.

Existing benchmarks like TIMEQA or TEMPREASON primarily focus on single-point timestamps or linear sequences, leaving a critical blind spot in AI evaluation. To address this, we introduce CoTempQA, a benchmark specifically designed to test co-temporal reasoning—the ability to process and compare multiple events happening simultaneously.

The CoTempQA Benchmark

CoTempQA consists of 4,748 samples derived from the Wikidata Knowledge Base (September 2023 dump). The dataset is structured around 17 predefined templates that convert structured Wikidata triples (subject, relation, object, start_time, end_time) into natural language questions.

To systematically evaluate model performance, we categorized co-temporal relationships into four distinct scenarios:

  1. Equal: Events that share identical start and end points.
  2. Overlap: Events that share a partial time segment.
  3. During: One event is entirely contained within the duration of another.
  4. Mix: Complex combinations involving multiple subjects or overlapping relations.

Unlike previous datasets that focus on isolated facts, CoTempQA requires models to perform interval comparisons, making it a significantly more challenging test of temporal logic.

Methodology: Mathematical Reasoning Chain of Thought (MR-CoT)

Our research highlights a significant finding: standard Chain-of-Thought (CoT) prompting, which encourages models to "think step by step," often fails or even degrades performance in co-temporal tasks. Models frequently hallucinate incorrect timelines or fail to logically intersect the intervals they have just described.

To combat this, we proposed MR-CoT (Math-Reasoning Chain of Thought). This framework shifts the model's approach from narrative reasoning to mathematical interval arithmetic through three steps:

  • Identify the Key Datetime: Extract the specific reference point or interval from the question.
  • Structure Timelines: Explicitly list the start and end dates for all entities involved.
  • Interval Intersection: Mathematically calculate the overlap between these intervals to derive the final answer.

Experimental Results and Insights

Our evaluation of state-of-the-art models in both Closed-Book (CBQA) and Open-Book (OBQA) settings revealed several critical insights:

The Human-AI Gap

There remains a staggering divide between AI and human performance. On the overall CoTempQA benchmark, humans achieve a score of 92.8, whereas the best-performing model, GPT-4, scores only 54.7 in the Open-Book setting. In Closed-Book settings, where models must rely on internal knowledge, GPT-4’s performance plummets to 14.5, suggesting that models struggle to retrieve and accurately compare precise dates from their training data.

The "Math-Time" Connection

Interestingly, models specialized in mathematics, such as WizardMath, outperformed much larger general-purpose models. This suggests that co-temporal reasoning is fundamentally a form of interval arithmetic rather than a purely linguistic task. By applying our MR-CoT strategy, we observed a 10.8 point improvement over standard baselines, confirming that treating time as a coordinate system is more effective than treating it as a narrative.

Complexity Bottlenecks

Models performed reasonably well on the "Equal" scenario (reaching 92.7% for GPT-4 in Open-Book) but struggled significantly as complexity increased. Performance dropped sharply in the "During" and "Mix" categories. Furthermore, models find it much harder to reason when multiple subjects are involved (e.g., comparing the timelines of two different people) than when a single subject is involved in multiple relations.

Conclusion and Future Work

CoTempQA exposes a significant limitation in the current generation of LLMs: the inability to handle concurrent temporal logic. While mathematical framing (MR-CoT) provides a path forward, the gap between AI and human reasoning remains wide.

Future research should focus on expanding the scale and diversity of co-temporal datasets, perhaps incorporating contextually rich sources like news archives. Additionally, improving Retrieval-Augmented Generation (RAG) tools will be essential for helping models fetch the precise temporal facts necessary for accurate reasoning in real-world applications. We hope CoTempQA serves as a foundational benchmark for the development of models that can truly "live in the moment."

Summary updated

Abstract

Temporal reasoning is fundamental for large language models (LLMs) to comprehend the world. Current temporal reasoning datasets are limited to questions about single or isolated events, falling short in mirroring the realistic temporal characteristics involving concurrent nature and intricate temporal interconnections. In this paper, we introduce CoTempQA, a comprehensive co-temporal Question Answering (QA) benchmark containing four co-temporal scenarios (Equal, Overlap, During, Mix) with 4,748 samples for evaluating the co-temporal comprehension and reasoning abilities of LLMs. Our extensive experiments reveal a significant gap between the performance of current LLMs and human-level reasoning on CoTempQA tasks. Even when enhanced with Chain of Thought (CoT) methodologies, models consistently struggle with our task. In our preliminary exploration, we discovered that mathematical reasoning plays a significant role in handling co-temporal events and proposed a strategy to boost LLMs' co-temporal reasoning from a mathematical perspective. We hope that our CoTempQA datasets will encourage further advancements in improving the co-temporal reasoning capabilities of LLMs. Our code is available at https://github.com/zhaochen0110/Cotempqa.
BibTeX
@inproceedings{su2024living,
  author = {Zhaochen Su and Juntao Li and Jun Zhang and Tong Zhu and Xiaoye Qu and Pan Zhou and Yan Bowen and Yu Cheng and Min Zhang},
  title = {{Living in the Moment: Can Large Language Models Grasp Co-Temporal Reasoning?}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
  url = {https://openreview.net/forum?id=9IVAa27rLG},
}
Figure