A Dataset for Answering Time-Sensitive Questions

Wenhu Chen, Xinyi Wang, William Yang Wang

TL;DR

This paper introduces TimeQA, a novel dataset designed to evaluate how well AI models can answer questions that depend on time-sensitive information. They found that current state-of-the-art models struggle significantly with temporal reasoning, achieving only 46% accuracy compared to humans' 87%, highlighting a critical gap in NLP capabilities.

What Problem Does This Paper Address?

As our world is dynamic, facts change over time – think of who the US President is or which team a famous athlete plays for. Existing question-answering (QA) datasets often lack questions that truly test a model's ability to understand and reason about these temporal shifts. Many datasets either have very few time-specific questions or their questions don't require actual temporal reasoning (e.g., they just copy time phrases from the text). This makes it hard to benchmark and improve models' temporal reasoning skills, which is crucial for many real-world applications.

Methods

The authors developed a multi-step pipeline to create the TimeQA dataset:

  1. Mining Time-Evolving Facts: They extracted facts with temporal information (start, end, or point in time) from Wikidata.
  2. Aligning Facts with Text: These mined facts were then linked to corresponding passages in Wikipedia.
  3. Human Verification: Crowd workers were employed to verify, correct, and calibrate these noisy fact-text pairs, ensuring accuracy and quality.
  4. Synthesizing Question-Answer Pairs: Using manually crafted templates and diverse reasoning types (like "in," "between," "before," "after"), they generated question-answer pairs. They created two versions: "easy" (explicit time specifiers) and "hard" (implicit time specifiers requiring more reasoning).

They also evaluated two state-of-the-art long-document QA models: BigBird (an extractive model) and FiD (a generative model).

Datasets

  • TimeQA: The primary contribution is this new dataset, containing approximately 20,000 question-answer pairs across "easy" and "hard" difficulty levels. It focuses on time-evolving facts extracted from Wikipedia.
  • Existing Datasets (for pre-training/fine-tuning): Natural Questions (NQ) and TriviaQA were used to pre-train and fine-tune the evaluation models.

Key Insights and Ideas

  • Temporal Reasoning is a Major Hurdle: Current advanced QA models, even those designed for long documents, perform poorly on time-sensitive questions, indicating a fundamental weakness in temporal reasoning.
  • Implicit Time is Harder: The "hard" version of TimeQA, which relies on implicit temporal information, significantly challenges models, showing that understanding context and inferring time is a key difficulty.
  • Human Performance is a High Bar: The substantial gap between model performance (46% on hard mode) and human performance (87%) underscores the need for new approaches to temporal reasoning.
  • FiD's Advantage in Long Documents: The FiD model, a generative approach, showed better performance and more consistent behavior across varying document lengths compared to BigBird, suggesting its architecture is more adept at handling long-term dependencies.

Key Results

  • Baseline Performance: Models fine-tuned on NQ/TriviaQA achieved only 33% (easy) and 27% (hard) accuracy on TimeQA.
  • Adaptive Fine-tuning: Fine-tuning these models on TimeQA boosted performance to 60% (easy) and 45% (hard).
  • Best Model Performance: The best-performing model, FiD, achieved 45.3% EM score on the hard version of TimeQA, far below the human benchmark of 87%.
  • Consistency Issues: Existing models show only 66% consistency when faced with minor perturbations in temporal specifiers, suggesting they are not robustly reasoning about time.
  • Implicit vs. Explicit: Models performed significantly better (64% vs. 49%) on questions with explicitly mentioned time specifiers compared to those with implicit ones.

Limitations and Future Work

  • Model Limitations: The paper clearly demonstrates that current models lack robust temporal reasoning capabilities, especially for implicit temporal information and consistent predictions.
  • Dataset Noise: While efforts were made to ensure quality, there's a possibility of residual noise in question or answer annotations.
  • Future Work: The authors suggest the need for developing new temporal-aware models that can better inject temporal commonsense knowledge. They also propose TimeQA as a benchmark for future research in this area, opening avenues for relation extraction and other temporal NLP tasks.

Summary updated

Abstract

Time is an important dimension in our physical world. Lots of facts can evolve with respect to time. For example, the U.S. President might change every four years. Therefore, it is important to consider the time dimension and empower the existing QA models to reason over time. However, the existing QA datasets contain rather few time-sensitive questions, hence not suitable for diagnosing or benchmarking the model's temporal reasoning capability. In order to promote research in this direction, we propose to construct a time-sensitive QA dataset. The dataset is constructed by 1) mining time-evolving facts from WikiData and aligning them to their corresponding Wikipedia page, 2) employing crowd workers to verify and calibrate these noisy facts, 3) generating question-answer pairs based on the annotated time-sensitive facts. Our dataset poses challenges in the aspect of both temporal understanding and temporal reasoning. We evaluate different SoTA long-document QA systems like BigBird and FiD on our dataset. The best-performing model FiD can only achieve 46\% accuracy, still far behind the human performance of 87\%. We demonstrate that these models are still lacking the ability to perform consistent temporal reasoning. Therefore, we believe that our dataset could serve as a benchmark to develop NLP models more sensitive to temporal shifts.
BibTeX
@inproceedings{chen2021dataset,
  author = {Wenhu Chen and Xinyi Wang and William Yang Wang},
  title = {{A Dataset for Answering Time-Sensitive Questions}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2021},
  url = {https://openreview.net/forum?id=9iEs_cngCBl},
}
Figure