TimeLitmus: A Diagnostic Benchmark for Cross-Modal Understanding and Explanation Faithfulness in Event-Conditioned Time-Series Prediction

Jie Gong, Maowei Jiang, Zhiwei Liu, Yankai Chen, Guojun Xiong, Xue Liu, Min Peng, Qianqian Xie, Sophia AnaniadouarXiv 2026 · arXiv

TL;DR

Large Language Models (LLMs) frequently achieve high accuracy on multimodal tasks combining text events and numerical time-series data, but are they actually understanding the connection? This paper introduces TimeLitmus, a diagnostic benchmark revealing that standard accuracy metrics mask massive failures in cross-modal reasoning and explanation faithfulness, with models relying heavily on superficial shortcuts rather than genuine data integration.

What Problem Does This Paper Address?

When an LLM predicts stock market reactions based on financial news and price history, or traffic delays based on an accident report and sensor data, a correct answer doesn't guarantee true understanding. The model might just be guessing based on the sentiment of the text (e.g., "downgrade" means prices go down) or blindly extrapolating a numerical trend, completely ignoring how the two modalities interact.

Furthermore, when asked to explain its reasoning, an LLM might generate a highly plausible explanation that has absolutely nothing to do with how it actually arrived at its answer—a phenomenon known as post-hoc rationalization.

Figure 1: Why accuracy alone is insufficient. Correct pre- dictions and plausible explanations may result from either genuine cross-modal integration or shortcut cues.

Existing benchmarks primarily measure task performance or answer correctness. They fail to diagnose whether correct outputs arise from genuine cross-modal integration or from unimodal cues and shallow statistical matching. The authors address this gap by creating a rigorous framework to test if LLMs genuinely integrate text and time-series data, and if their explanations are faithful to their internal mechanics.

Methods

The authors design TimeLitmus around controlled, counterfactual interventions. Instead of evaluating models on isolated data points, they test them on paired variants to see if the model's predictions change consistently with targeted modifications:

  • Series-side counterfactuals: Keeping the textual event identical but altering the pre-event time-series state.
  • Event-side counterfactuals: Keeping the time-series history identical but modifying key semantic details of the text event.
  • Hard Paired Contrasts (HPC): Evaluating natural pairs that look highly similar on the surface but require completely different predictions.
  • Shortcut Controls: Stripping away one modality (event-only or series-only inputs) or adding irrelevant cues to isolate whether the model is exploiting shallow patterns.
  • Explanation Faithfulness Testing: Measuring the Claim-Supported Dependence Rate (CSDR). If a model's explanation claims it relied on a specific factor (like a sudden speed drop), the benchmark intervenes on that exact factor to see if the model's prediction behavior actually responds accordingly.
Figure 2: Overview of TimeLitmus. Controlled interventions and faithfulness diagnostics test whether event-conditioned predictions and explanations rely on the intended cross- modal evidence rather than shortcuts.

Datasets

TimeLitmus introduces a benchmark of 4,856 evaluation records split across two high-stakes domains:

  1. Finance: Built on the FNSPID dataset, mapping financial news events to historical equity variables (normalized closing prices, daily returns, volume).
  2. Traffic: Built on the TraffiDent dataset, aligning roadway incident descriptions with pre-report sensor histories (speed, flow, occupancy).

The benchmark is structured into four diagnostic families: Natural Prediction, Correctness Attribution, Explanation Faithfulness, and Shortcut Robustness. Additionally, the authors provide a separate 3,000-instance natural prediction corpus used exclusively for supervised fine-tuning (QLoRA) experiments.

Key Insights and Ideas

  • The Illusion of Competence: High standard accuracy on natural data is often a mirage. Models that look capable fall apart under minor counterfactual tweaks, exposing a reliance on unimodal shortcuts.
  • Saying One Thing, Doing Another: LLMs are excellent at mentioning the right variables in their text explanations (often achieving a >90% citation rate), but their underlying prediction behavior shows they aren't actually using that information to drive decisions.
  • Recognition vs. Application: When presented with two scenarios jointly, models can explicitly recognize the relationship between them. Yet, they fail to apply this exact same logic when making independent predictions.
  • Fine-Tuning Isn't a Quick Fix: Training models on standard natural prediction data (via QLoRA) makes them better at selecting evidence and increases confidence-level sensitivity, but it completely fails to improve their robust cross-modal reasoning or explanation faithfulness.

Key Results

  • Low Pair Correctness: While standard accuracy looks decent, Hard Paired Contrast (HPC) correctness peaks at a dismal 19.2% in Finance and 11.7% in Traffic across ten representative LLMs.
  • Shortcut Reliance: In Traffic series-side controls, series-only inputs frequently matched or exceeded full-input accuracy across models, proving a heavy reliance on historical trends while ignoring the event.
  • Faithfulness Gap: Stated explanations frequently lack behavioral support. In the Traffic domain, behavioral support for cited factors remained below 22% despite high citation rates.
  • Humans Outperform LLMs: Human annotators easily outperformed the LLM envelope across all dimensions, scoring 53.3% on Finance series-side counterfactuals compared to the best LLM's 20.0%, confirming that these cross-modal distinctions are fully recoverable from the inputs.
Figure 5: Paired QLoRA-minus-Base effects across scales and domains. Natural-only adaptation produces selective gains concentrated in evidence selection, distinct from transfer to controlled and hard-pair behavior.

Limitations and Future Work

The authors acknowledge that standard supervised adaptation (natural-only QLoRA) yields highly localized improvements (mostly in evidence selection) and does not automatically transfer to robust, counterfactual consistency or explanation faithfulness.

Future work must explore alternative training paradigms that explicitly enforce cross-modal relational alignment and penalize unfaithful post-hoc rationalizations, ensuring that models are correct for the right reasons.

Summary updated

Abstract

Large language models (LLMs) are increasingly used to make predictions from numerical time-series histories and textual events. Yet accuracy alone cannot reveal whether correct answers reflect effective integration of the two inputs or instead arise from event polarity, unimodal priors, or superficial cues. Likewise, plausible explanations may rationalize predictions without faithfully reflecting the evidence that drives model behavior. We introduce TimeLitmus, a diagnostic benchmark for cross-modal understanding and explanation faithfulness in event-conditioned time-series prediction. TimeLitmus contains 4,856 evaluation records across Finance and Traffic, combining natural prediction with controlled counterfactual and contrastive interventions, explanation-targeted faithfulness tests, and systematic shortcut controls. Across ten representative LLMs, standard prediction accuracy substantially overstates reliable cross-modal understanding: Hard Paired Contrast (HPC) pair correctness peaks at only 19.2% in Finance and 11.7% in Traffic, and all ten models show lower-than-expected consistency on Finance series-side controls. Models often recognize scenario relations explicitly yet fail to apply them during independent prediction. Explanation faithfulness shows a similar gap: in Traffic, most models cite the manipulated temporal factor in over 90% of cases, while behavioral support remains below 22%. Human annotators outperform LLMs on matched controlled and hard-pair diagnostics, confirming that these distinctions are recoverable from the inputs. Natural-only adaptation yields selective gains in evidence selection and input sensitivity, but not consistent gains in controlled or hard-pair behavior. The benchmark, evaluation suite, and supervised adaptation data will be released publicly.
BibTeX
@misc{gong2026timelitmus,
  author = {Jie Gong and Maowei Jiang and Zhiwei Liu and Yankai Chen and Guojun Xiong and Xue Liu and Min Peng and Qianqian Xie and Sophia Ananiadou},
  title = {{TimeLitmus: A Diagnostic Benchmark for Cross-Modal Understanding and Explanation Faithfulness in Event-Conditioned Time-Series Prediction}},
  year = {2026},
  eprint = {2609.24677},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2609.24677},
}
Figure