This paper puts the recent wave of Time-Series Foundation Models (TSFMs) to the ultimate test: forecasting the notoriously "unpredictable" intraday returns of liquid index futures. In a rigorous, pre-registered study, the authors find that while these models look impressive on general benchmarks, they provide zero incremental skill over a simple random walk when applied to financial returns.
TL;DR
In a pre-registered blind evaluation of six major time-series foundation models (including Chronos and TimesFM), none were able to outperform a simple "no-change" forecast for intraday index-futures returns. The study reveals that apparent model "skill" often stems from metric artifacts or simple input ingredients rather than actual predictive power.
What Problem Does This Paper Address?
Time-series foundation models have shown remarkable zero-shot capabilities across diverse domains. Recently, finance-specific models like Kronos have reported significant gains in forecasting financial data. However, financial returns are characterized by an extremely low signal-to-noise ratio, making them a "hard mode" for any forecaster.
The authors address a critical question: Is the reported success of TSFMs in finance due to genuine predictive skill, or is it an illusion created by choice of metrics, lack of hard benchmarks (like the random walk), or data leakage? To answer this, they employ a "blind" evaluation protocol—testing models on data they couldn't have seen, using a fixed analysis plan locked before the results were known.
Methods
The study evaluates a panel of five frozen zero-shot models—Chronos-Bolt, Chronos-2, TimesFM 2.5, Moirai 2.0, and TiRex—plus experiments with Kronos.
Key methodological pillars include:
- Pre-registration: The analysis plan, including models, metrics, and "pass" criteria, was hashed and timestamped before opening the test data to prevent p-hacking.
- Input Representations: Models were tested with four different inputs: native prices (R0), log returns (R1), volatility-standardized returns (R2), and a volume-based clock (R4).
- Hard Benchmarks: The primary competitor was the "no-change" forecast ($f=0$), representing the random walk hypothesis. They also used a transformation-aware ridge regression as a control.
- Planted Signals: To ensure the test could detect a signal if one existed, the authors created synthetic series with known predictability to establish detection limits.
Datasets
The evaluation focused on two highly liquid markets:
- E-mini Nasdaq-100 (NQ) futures (Primary market)
- E-mini S&P 500 (ES) futures (Replication market)
The data was split into:
- Train/Validation: 2018–2024.
- Blind Set: January 2025 – May 2026 (opened once after freezing the plan).
- Hold-out ("Ultra-blind"): September – October 2026 (a final descriptive check).
Key Insights and Ideas
- The RankIC Illusion: One of the most striking findings is that standard metrics like RankIC (rank correlation) can be highly misleading. The authors prove that a trivial "revert-to-mean" rule yields a positive RankIC under a random walk due to the way data is centered within a day. Figure 3 illustrates how a sign-randomized null reproduces ~95% of the observed "skill" in this metric.
- Ingredient Controls: When a model appeared to beat a volatility benchmark (HAR), the authors found the gain was entirely explained by the Exponential Weighted Moving Average (EWMA) already present in the model's input. The model itself added no incremental value.
- Magnitude vs. Accuracy: Changing input representations (like log returns vs. native prices) changed the magnitude of the models' forecasts but did not improve their accuracy.
- Detection Limits: The models could recover strong synthetic signals at a 5-minute horizon, but this ability deteriorated rapidly by 30 minutes. This suggests that for real-world returns, the signal is simply below the models' current resolution.
Key Results
The headline result is a "bounded negative":
- 0 out of 160 model-representation-horizon-market cells beat the no-change forecast on the blind set.
- Hansen’s SPA test yielded a p-value of 1.000 for both markets, indicating no evidence of superior predictive ability.
- As shown in Figure 1, every single Mean Absolute Error (MAE) ratio was above 1.0, meaning the models were consistently less accurate than predicting zero change.
- Price-level correlation exceeded 0.9996 for all models, but the no-change forecast scored even higher, proving that high correlation in price levels is a meaningless metric for forecasting skill.
Limitations and Future Work
- Scope: The study is limited to two index futures over a specific timeframe (2023–2026). Results might differ for less liquid assets or different regimes.
- Frozen Models: The study primarily evaluated "frozen" zero-shot models. While some fine-tuning was tested with Kronos (and also failed), more extensive domain-specific tuning remains a potential avenue.
- Input Data: The authors noted that without signed trade data (order flow), certain high-frequency signals were not testable.
- Future Direction: The authors recommend that future financial AI research must use "no-change" as a baseline, calibrate rank metrics against sign-randomized nulls, and explicitly state the detection limits of their study.
Summary updated