Beyond Naïve Prompting: Strategies for Improved Zero-shot Context-aided Forecasting with LLMs

Arjun Ashok, Andrew Robert Williams, Vincent Zhihao Zheng, Irina Rish, Nicolas Chapados, Étienne Marcotte, Valentina Zantedeschi, Alexandre Drouin

TL;DR

This paper introduces a framework of four strategies to improve the accuracy and efficiency of Large Language Models (LLMs) for context-aided time series forecasting. They address critical limitations like lack of interpretability and high computational costs, showing significant performance gains and offering practical solutions for real-world applications.

What Problem Does This Paper Address?

Forecasting future trends based on historical data is crucial for decision-making. However, real-world forecasting also heavily relies on contextual information, often available in natural language. While Large Language Models (LLMs) show promise in incorporating this context, current methods are often "naïve" – they struggle with interpretability (understanding why a forecast is wrong), achieve suboptimal accuracy, and are computationally expensive. This paper aims to bridge this gap by developing strategies that make LLM-based forecasting more diagnostic, accurate, and efficient.

Methods

The authors propose a unified framework of four complementary strategies that extend basic "Direct Prompting" (DP) for LLMs:

  • FxDP (Forecast Effect Explanation): This strategy elicits explanations from the LLM about how the context influences the forecast before generating the forecast itself. This allows for diagnosing failure modes, specifically identifying an "Execution Gap" where models understand the context but fail to apply it.
  • CorDP (Forecast Correction): Instead of forecasting from scratch, CorDP uses LLMs to correct existing probabilistic forecasts from a quantitative model based on textual context. This preserves the accuracy of specialized models while adding contextual reasoning.
  • IC-DP (In-Context Direct Prompting): This method leverages in-context learning by providing the LLM with a few examples of context-aided forecasting tasks within the prompt. This helps the LLM learn the mapping from context to forecast effect with minimal examples.
  • RouteDP (Direct Prompt with Model Routing): To address computational costs, RouteDP uses a router to assess task difficulty and intelligently assign tasks to either a small, efficient LLM or a larger, more capable LLM. This balances performance and efficiency under compute constraints.

Datasets

The primary dataset used for evaluation is the Context-Is-Key (CiK) benchmark. CiK comprises 2,644 time series across 7 real-world domains, specifically designed to benchmark context-aided forecasting models where textual context is necessary for accurate forecasts. The tasks are designed to mitigate memorization effects.

Key Insights and Ideas

  • The "Execution Gap": A significant finding is that LLMs often correctly explain how context affects a forecast but fail to apply this reasoning to produce an improved forecast. This "Execution Gap" persists even in frontier models, highlighting a fundamental challenge in translating understanding into action.
  • Model Size Matters, But Not Exclusively: While larger models generally perform better, especially in generating accurate explanations (FxDP), smaller models can achieve competitive accuracy when augmented with strategies like CorDP and IC-DP, especially when leveraging strong base forecasters.
  • Complementarity of Strategies: The proposed strategies are orthogonal and can be combined to achieve further improvements. For instance, combining IC-DP with CorDP or using RouteDP with IC-DP/CorDP can yield better results than individual methods.
  • Practicality is Key: CorDP offers a practical way to integrate LLMs into existing forecasting pipelines by correcting, rather than replacing, specialized quantitative models, leading to substantial gains with minimal overhead.

Key Results

  • Accuracy Improvements: CorDP achieved improvements of up to 50% over Direct Prompting. IC-DP improved performance for 14/16 models, with gains of 14-56% for small models and 20-40% for larger ones.
  • Efficiency Gains: RouteDP demonstrated substantial improvements in forecast accuracy (up to 46%) at a fraction of the cost by intelligently routing tasks.
  • Diagnostic Power: FxDP revealed the "Execution Gap," showing that models can explain context effects but fail to apply them, a pattern that scales with model size but persists across all scales.
  • LLM Judge Reliability: The study validated the use of LLM judges for evaluating forecast effect explanations, showing substantial agreement with human evaluators.

Limitations and Future Work

  • Execution Gap Persistence: The paper acknowledges the persistent "Execution Gap" as a fundamental challenge, suggesting that larger models require enhanced ability to translate explanations into improved forecasts, while smaller models need better reasoning capabilities.
  • LLM Orchestration: Future work could explore LLMs as orchestrators or agents to dynamically select forecasting techniques or adjust outputs, and investigate the computational cost-benefit of such approaches.
  • Training-Based Methods: Expanding strategies to training-based methods could allow for trained routers in RouteDP or models that leverage arbitrary base forecasters in CorDP.
  • Unconstrained Settings: Evaluating strategies in more unconstrained settings, where context might be irrelevant or excessively long, is an interesting future direction.
  • Cost Barrier: The high cost of LLMs compared to traditional forecasting methods remains a barrier to deployment. Future research should focus on developing more efficient context-aided forecasting models from the ground up.

Summary updated

BibTeX
@inproceedings{ashok2025beyond,
  author = {Arjun Ashok and Andrew Robert Williams and Vincent Zhihao Zheng and Irina Rish and Nicolas Chapados and Étienne Marcotte and Valentina Zantedeschi and Alexandre Drouin},
  title = {{Beyond Naïve Prompting: Strategies for Improved Zero-shot Context-aided Forecasting with LLMs}},
  booktitle = {BERT2S @ NeurIPS},
  year = {2025},
  url = {https://openreview.net/forum?id=fgcNDgZVfW},
}
Figure