Time to Revist Exact Match

Auss Abbood, Zaiqiao Meng, Nigel CollierarXiv 2025 · DOI · arXiv

Introduction

Standard evaluation for Large Language Models (LLMs) often relies on Exact Match (EM), a binary metric that treats any deviation from the ground truth as a total failure. In the context of temporal reasoning—where answers are numerical (e.g., "How long is a flight from London to New York?")—this approach is fundamentally flawed. EM cannot distinguish between a minor one-hour error and a catastrophic ten-year hallucination. This paper argues for a shift toward regression-based metrics that account for the magnitude and plausibility of temporal mistakes.

Comparison of Exact Match vs. Temporal Difference evaluation.

Methodology: Beyond Binary Matching

To address the limitations of EM, we propose framing temporal QA as a numerical estimation task. We move away from binary text matching and toward forecasting metrics:

  • sMAPE (symmetric Mean Absolute Percentage Error): A scale-free metric that measures the percentage error of a prediction. It allows for comparison across different units (e.g., comparing an error in seconds to an error in years).
  • MASE (Mean Absolute Scaled Error): This scales the model's error by the average deviation found in the ground truth data. It assesses whether an error is reasonable given the typical range of answers for a specific topic.

A specialized parsing pipeline was developed to convert raw LLM text responses into "time-aware objects" (integers, timedeltas, or datetime objects) that can be processed mathematically.

The TempAnswerQA Benchmark

We introduce TempAnswerQA, a benchmark specifically designed for numerical temporal evaluation. Distilled from the Test of Time (ToT) and TempTabQA (TTQA) datasets, it contains 3,434 QA pairs where every question requires a numeric temporal answer (dates or durations). The benchmark includes real-world questions from Wikipedia tables and synthetic reasoning tasks involving arithmetic and semantics.

Key Insights

Our analysis reveals several critical phenomena in how LLMs handle time:

  • The ±1 Phenomenon: The most frequent error across all models was deviating by exactly ±1 from the ground truth. This is often due to "transitional time" ambiguities—like whether to include the start or end date in a duration calculation.
  • Directional Bias: Errors are not symmetric. We found that "positive" errors (overestimations) were significantly larger in magnitude than underestimations.
  • The Synthetic Gap: MASE revealed that models trained heavily on synthetic data (like the Qwen and Phi series) often performed worse when errors were scaled by domain plausibility. They may excel at arithmetic but lack the "common sense" of realistic temporal ranges.

Results: The Ranking Reshuffle

Evaluating models with sMAPE and MASE leads to significant shifts in performance rankings compared to traditional EM.

Model ranking shifts when moving from Exact Match to sMAPE.

As shown in the figure above, Qwen2.5-14B overtakes Llama-3.3-70B for the top spot when using sMAPE, despite Llama winning on the EM metric. Conversely, Llama-3.1-8B shows remarkable robustness; while it misses the "exact" string more often, its guesses are much more plausible within the temporal domain, causing it to jump from last place to second on certain MASE-evaluated splits.

Conclusion

Exact Match is an insufficient proxy for the utility and safety of temporal reasoning in LLMs. By adopting metrics like sMAPE and MASE, we gain a more nuanced understanding of model performance, identifying those that are "close enough" for practical deployment versus those prone to high-risk hallucinations. Future work should focus on handling non-numeric temporal responses and integrating tool-calling to mitigate the arithmetic errors that currently plague these models.

Summary updated

Abstract

Temporal question answering is an established method for evaluating temporal reasoning in large language models. Expected answers are often numeric (e.g., dates or durations), yet model responses are evaluated like regular text with exact match (EM), unable to distinguish small from large errors. In this investigative work, we frame temporal question answering as a numerical estimation task to assess the shortcomings of EM. We introduce TempAnswerQA, a benchmark distilled from Test of Time and TempTabQA, where all questions require a numerical, temporal answer, allowing us to evaluate models beyond EM. We use the forecasting metrics symmetric mean absolute percentage error (sMAPE) and mean absolute scaled error (MASE). With sMAPE, we find that error size and EM are decoupled. Models with low EM still have low sMAPE (both ~20%), and some models have high sMAPE despite high EM. Scaling errors by the deviation of the ground truth data with MASE reshuffles model rankings compared to EM, revealing gaps in models' understanding of temporal domain knowledge, especially when trained with synthetic data. Lastly, the models' most frequent error is to deviate by only $\pm1$ from the ground truth. sMAPE and MASE, unlike EM, adequately weight these errors. Our findings underscore the need for specialised metrics for temporal QA tasks. Code and data are available on https://github.com/aauss/temporal-answer-qa.
BibTeX
@misc{abbood2025time,
  author = {Auss Abbood and Zaiqiao Meng and Nigel Collier},
  title = {{Time to Revist Exact Match}},
  year = {2025},
  eprint = {2509.16720},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2509.16720},
}
Figure