Introduction
Standard evaluation for Large Language Models (LLMs) often relies on Exact Match (EM), a binary metric that treats any deviation from the ground truth as a total failure. In the context of temporal reasoning—where answers are numerical (e.g., "How long is a flight from London to New York?")—this approach is fundamentally flawed. EM cannot distinguish between a minor one-hour error and a catastrophic ten-year hallucination. This paper argues for a shift toward regression-based metrics that account for the magnitude and plausibility of temporal mistakes.
Methodology: Beyond Binary Matching
To address the limitations of EM, we propose framing temporal QA as a numerical estimation task. We move away from binary text matching and toward forecasting metrics:
- sMAPE (symmetric Mean Absolute Percentage Error): A scale-free metric that measures the percentage error of a prediction. It allows for comparison across different units (e.g., comparing an error in seconds to an error in years).
- MASE (Mean Absolute Scaled Error): This scales the model's error by the average deviation found in the ground truth data. It assesses whether an error is reasonable given the typical range of answers for a specific topic.
A specialized parsing pipeline was developed to convert raw LLM text responses into "time-aware objects" (integers, timedeltas, or datetime objects) that can be processed mathematically.
The TempAnswerQA Benchmark
We introduce TempAnswerQA, a benchmark specifically designed for numerical temporal evaluation. Distilled from the Test of Time (ToT) and TempTabQA (TTQA) datasets, it contains 3,434 QA pairs where every question requires a numeric temporal answer (dates or durations). The benchmark includes real-world questions from Wikipedia tables and synthetic reasoning tasks involving arithmetic and semantics.
Key Insights
Our analysis reveals several critical phenomena in how LLMs handle time:
- The ±1 Phenomenon: The most frequent error across all models was deviating by exactly ±1 from the ground truth. This is often due to "transitional time" ambiguities—like whether to include the start or end date in a duration calculation.
- Directional Bias: Errors are not symmetric. We found that "positive" errors (overestimations) were significantly larger in magnitude than underestimations.
- The Synthetic Gap: MASE revealed that models trained heavily on synthetic data (like the Qwen and Phi series) often performed worse when errors were scaled by domain plausibility. They may excel at arithmetic but lack the "common sense" of realistic temporal ranges.
Results: The Ranking Reshuffle
Evaluating models with sMAPE and MASE leads to significant shifts in performance rankings compared to traditional EM.
As shown in the figure above, Qwen2.5-14B overtakes Llama-3.3-70B for the top spot when using sMAPE, despite Llama winning on the EM metric. Conversely, Llama-3.1-8B shows remarkable robustness; while it misses the "exact" string more often, its guesses are much more plausible within the temporal domain, causing it to jump from last place to second on certain MASE-evaluated splits.
Conclusion
Exact Match is an insufficient proxy for the utility and safety of temporal reasoning in LLMs. By adopting metrics like sMAPE and MASE, we gain a more nuanced understanding of model performance, identifying those that are "close enough" for practical deployment versus those prone to high-risk hallucinations. Future work should focus on handling non-numeric temporal responses and integrating tool-calling to mitigate the arithmetic errors that currently plague these models.
Summary updated