Abstract: Temporal question answering is an established method for evaluating temporal reasoning in large language models. Expected answers are often numeric (e.g., dates or durations), yet model responses are evaluated like regular text with exact match (EM), unable to distinguish small from large errors. In this investigative work, we frame temporal question answering as a numerical estimation task to assess the shortcomings of EM. We introduce TempAnswerQA, a benchmark distilled from Test of Time and TempTabQA, where all questions require a numerical, temporal answer, allowing us to evaluate models beyond EM. We use the forecasting metrics symmetric mean absolute percentage error (sMAPE) and mean absolute scaled error (MASE). With sMAPE, we find that error size and EM are decoupled. Models with low EM still have low sMAPE (both 20%), and some models have high sMAPE despite high EM. Scaling errors by the deviation of the ground truth data with MASE reshuffles model rankings compared to EM, revealing gaps in models’ understanding of temporal domain knowledge, especially when trained with synthetic data. Lastly, the models’ most frequent error is to deviate by only ±1 from the ground truth. sMAPE and MASE, unlike EM, adequately weight these errors. Our findings underscore the need for specialised metrics for temporal QA tasks.
The Limitations of Exact Match in Temporal QA
In temporal Question Answering (QA), models are frequently asked to provide numerical values, such as "How many hours before anesthesia should you stop taking Levodopa?" or "What year did this event occur?" Currently, these tasks are evaluated using Exact Match (EM)—a binary string-matching metric. If the ground truth is 8 and the model responds with 9, it receives a score of zero, the same penalty it would receive for responding with 800.
As illustrated above, EM is fundamentally flawed for temporal reasoning for three primary reasons:
- Continuous Nature of Time: Time is numerical. Error magnitudes matter in real-world applications like medical dosages or logistics.
- Transitional Ambiguity: Questions about age or duration often have a ±1 margin of error due to inclusive/exclusive date logic, yet EM penalizes these "near-misses" as total failures.
- Approximation: LLMs often approximate numerical answers. EM ignores models that are "close enough" to being correct.
Framing Temporal QA as Numerical Estimation
To address these shortcomings, the researchers frame temporal QA as a numerical estimation task. They move away from string matching toward regression-based metrics borrowed from forecasting:
- sMAPE (Symmetric Mean Absolute Percentage Error): A scale-free metric measuring percentage error, allowing for comparison across different units (e.g., seconds vs. years).
- MASE (Mean Absolute Scaled Error): This scales the model's error against typical variation in the ground-truth data, assessing if an error is "reasonable" given the domain context.
To facilitate this, the authors developed a pipeline to parse raw model text into "time-aware" Python objects (integers, timedeltas, or datetimes) for mathematical comparison. They introduced TempAnswerQA, a benchmark of 3,434 QA pairs distilled from Test of Time (synthetic arithmetic) and TempTabQA (real-world Wikipedia data).
Key Results and Ranking Shifts
The study reveals that EM and error magnitude are largely decoupled. A model can achieve a high EM score while still producing massive, hallucinatory errors when it misses. Conversely, models with low EM are often highly accurate but frequently "off-by-one." In the TempTabQA dataset, nearly 50% of all errors were exactly ±1.
When evaluated by sMAPE, Qwen2.5-14B overtook Llama-3.3-70B as the top performer, despite Llama's higher EM score. Furthermore, Llama-3.1-8B jumped from last place to second place when evaluated by MASE, indicating that while it misses the "exact" answer often, its guesses are much more plausible within the context of the question than those of other models.
Conclusion
The findings suggest that synthetic training data may help models "hit the bullseye" on specific patterns (improving EM) but can distort their broader temporal common sense. The researchers advocate for the adoption of specialized metrics like sMAPE and MASE to provide a more nuanced and truthful evaluation of temporal reasoning in LLMs.
Summary updated