TL;DR
Large Language Models (LLMs) frequently achieve high accuracy on multimodal tasks combining text events and numerical time-series data, but are they actually understanding the connection? This paper introduces TimeLitmus, a diagnostic benchmark revealing that standard accuracy metrics mask massive failures in cross-modal reasoning and explanation faithfulness, with models relying heavily on superficial shortcuts rather than genuine data integration.
What Problem Does This Paper Address?
When an LLM predicts stock market reactions based on financial news and price history, or traffic delays based on an accident report and sensor data, a correct answer doesn't guarantee true understanding. The model might just be guessing based on the sentiment of the text (e.g., "downgrade" means prices go down) or blindly extrapolating a numerical trend, completely ignoring how the two modalities interact.
Furthermore, when asked to explain its reasoning, an LLM might generate a highly plausible explanation that has absolutely nothing to do with how it actually arrived at its answer—a phenomenon known as post-hoc rationalization.
Existing benchmarks primarily measure task performance or answer correctness. They fail to diagnose whether correct outputs arise from genuine cross-modal integration or from unimodal cues and shallow statistical matching. The authors address this gap by creating a rigorous framework to test if LLMs genuinely integrate text and time-series data, and if their explanations are faithful to their internal mechanics.
Methods
The authors design TimeLitmus around controlled, counterfactual interventions. Instead of evaluating models on isolated data points, they test them on paired variants to see if the model's predictions change consistently with targeted modifications:
- Series-side counterfactuals: Keeping the textual event identical but altering the pre-event time-series state.
- Event-side counterfactuals: Keeping the time-series history identical but modifying key semantic details of the text event.
- Hard Paired Contrasts (HPC): Evaluating natural pairs that look highly similar on the surface but require completely different predictions.
- Shortcut Controls: Stripping away one modality (event-only or series-only inputs) or adding irrelevant cues to isolate whether the model is exploiting shallow patterns.
- Explanation Faithfulness Testing: Measuring the Claim-Supported Dependence Rate (CSDR). If a model's explanation claims it relied on a specific factor (like a sudden speed drop), the benchmark intervenes on that exact factor to see if the model's prediction behavior actually responds accordingly.
Datasets
TimeLitmus introduces a benchmark of 4,856 evaluation records split across two high-stakes domains:
- Finance: Built on the FNSPID dataset, mapping financial news events to historical equity variables (normalized closing prices, daily returns, volume).
- Traffic: Built on the TraffiDent dataset, aligning roadway incident descriptions with pre-report sensor histories (speed, flow, occupancy).
The benchmark is structured into four diagnostic families: Natural Prediction, Correctness Attribution, Explanation Faithfulness, and Shortcut Robustness. Additionally, the authors provide a separate 3,000-instance natural prediction corpus used exclusively for supervised fine-tuning (QLoRA) experiments.
Key Insights and Ideas
- The Illusion of Competence: High standard accuracy on natural data is often a mirage. Models that look capable fall apart under minor counterfactual tweaks, exposing a reliance on unimodal shortcuts.
- Saying One Thing, Doing Another: LLMs are excellent at mentioning the right variables in their text explanations (often achieving a >90% citation rate), but their underlying prediction behavior shows they aren't actually using that information to drive decisions.
- Recognition vs. Application: When presented with two scenarios jointly, models can explicitly recognize the relationship between them. Yet, they fail to apply this exact same logic when making independent predictions.
- Fine-Tuning Isn't a Quick Fix: Training models on standard natural prediction data (via QLoRA) makes them better at selecting evidence and increases confidence-level sensitivity, but it completely fails to improve their robust cross-modal reasoning or explanation faithfulness.
Key Results
- Low Pair Correctness: While standard accuracy looks decent, Hard Paired Contrast (HPC) correctness peaks at a dismal 19.2% in Finance and 11.7% in Traffic across ten representative LLMs.
- Shortcut Reliance: In Traffic series-side controls, series-only inputs frequently matched or exceeded full-input accuracy across models, proving a heavy reliance on historical trends while ignoring the event.
- Faithfulness Gap: Stated explanations frequently lack behavioral support. In the Traffic domain, behavioral support for cited factors remained below 22% despite high citation rates.
- Humans Outperform LLMs: Human annotators easily outperformed the LLM envelope across all dimensions, scoring 53.3% on Finance series-side counterfactuals compared to the best LLM's 20.0%, confirming that these cross-modal distinctions are fully recoverable from the inputs.
Limitations and Future Work
The authors acknowledge that standard supervised adaptation (natural-only QLoRA) yields highly localized improvements (mostly in evidence selection) and does not automatically transfer to robust, counterfactual consistency or explanation faithfulness.
Future work must explore alternative training paradigms that explicitly enforce cross-modal relational alignment and penalize unfaithful post-hoc rationalizations, ensuring that models are correct for the right reasons.
Summary updated