EviCast: Forecasting Retrieval Gains Through Shared Response Calibration

Yibo Wang, Jisheng Dang, Bimei Wang, Jiongpei Hou, Jingwen Zhao, Wenjun Ma, Fei Shen, Juncheng Li, Meng Wang, Qi Tian, Bin Hu, Tat-Seng ChuaarXiv 2026 · arXiv

TL;DR

EviCast is a new framework that predicts the performance gains of Retrieval-Augmented Generation (RAG) configurations before you deploy them. By decoupling how a model reacts to specific types of evidence from how often a retriever finds that evidence, it can diagnose multiple retrieval strategies with up to 80% fewer LLM calls than traditional testing.

What Problem Does This Paper Address?

Deploying RAG is often a gamble. While adding external knowledge should help, it frequently doesn't. Irrelevant or distracting context can actually degrade a Large Language Model's (LLM) performance, a phenomenon often called "distraction" or "lost in the middle." Furthermore, retrieval isn't free—it adds significant latency and inference costs.

Currently, engineers face a "deployment dilemma": do you indiscriminately enable retrieval for everything, or do you spend massive amounts of compute testing every possible retriever configuration (dense, sparse, hybrid, etc.) on your full dataset to see which one works? Existing evaluation tools like RAGAS or RAGChecker are "post-hoc"—they tell you how your system performed after you've already spent the money to run it. EviCast aims to provide a "forecast" to help decide which configurations are worth the investment upfront.

The deployment dilemma: balancing performance risk against the high cost of testing every configuration.

Methods

The core innovation of EviCast is interventional evidence-state decomposition. It breaks down the "utility" of a RAG setup into two independent parts:

  1. Shared Response Calibration: You take a small calibration set and "intervene" by forcing the LLM to generate answers based on specific evidence states (e.g., "here is a perfect answer," "here is a misleading distractor," or "here is a bridge passage but no final answer"). This creates a profile of how the generator reacts to different structures of information. Crucially, this is done only once per generator/task.
  2. Retriever Characterization: You run your candidate retrievers offline on the same small calibration set. You don't need the LLM here; you just count how often each retriever produces those specific evidence states.

By combining these two, EviCast forecasts the expected gain (net utility) for any number of retrievers without needing to run the LLM for every single one.

Fig. 2. EviCast framework for reusable retrieval utility forecasting and deployment.

The authors propose two versions:

  • EviCast-B (Binary): Simple "supportive" vs. "misleading" states.
  • EviCast-T (Topology-aware): Designed for multihop reasoning, it distinguishes between Terminal evidence (contains the answer) and Bridge evidence (connects the reasoning chain).

Finally, they use a Two-Gate Decision Rule. A RAG configuration is only enabled if it passes a "separability gate" (does the retriever actually distinguish relevant info from noise?) and a "utility gate" (is the predicted gain statistically significant?).

Datasets

The authors validated EviCast across a diverse range of tasks and models:

  • Multihop QA: HotpotQA, MuSiQue, and 2WikiMultiHopQA.
  • Emotion Recognition: EMER, MELD, and RAVDESS (used to test cross-task generalization).
  • Models: Qwen2.5 (3B and 7B scales) and Phi-4-multimodal.
  • Retrievers: A mix of dense (BGE-M3), sparse (BM25, TF-IDF), and fusion (RRF) methods.

Key Insights and Ideas

  • Retrievability ≠ Utility: Just because a retriever is good at finding relevant documents doesn't mean the LLM is good at using them. EviCast treats these as two separate variables.
  • Topology is King in Multihop: In complex reasoning, not all "correct" passages are equal. A bridge passage provides a different utility than a terminal passage. EviCast-T shows that distinguishing these roles is vital for accurate forecasting.
  • Statistical Efficiency: Because the LLM's response to evidence structure is relatively stable for a given task, you don't need a huge sample to calibrate it. This allows EviCast to be much more efficient than "Direct" evaluation (testing every retriever-generator pair).
Fig. 3. Two-gate decision plane for 10 cross-task configurations.

Key Results

  • High Accuracy: On HotpotQA, EviCast correctly decided all 14 retrieval configurations with a forecast error of only 2.75 F1 points.
  • Massive Savings: It reduced diagnostic generation calls by 80% compared to direct testing.
  • Superior Forecasting: In low-budget regimes (where you can only afford a few LLM calls), EviCast outperformed seven other state-of-the-art performance predictors.
  • Robustness: The framework successfully transferred its findings across different datasets and model scales, maintaining high decision agreement (90% across 10 diverse task configurations).
Fig. 5. Forecast and independent test utility for 14 HotpotQA retrieval configurations, with EviCast on the left and Direct calibration on the right. Error bars show 95% CIs.

Limitations and Future Work

While powerful, EviCast currently has some boundaries:

  • Fixed Depth: The current experiments primarily focus on top-2 retrieval results. Extending this to deeper retrieval (top-k where k is large) or iterative RAG remains a future challenge.
  • Calibration Labels: The framework requires a small set of labeled data (knowing which passages are "terminal" or "bridge") to perform the initial calibration.
  • Future Directions: The authors suggest extending evidence states to handle multimodal environments and more complex, variable-depth retrieval chains.

Summary updated

Abstract

Retrieval-augmented generation (RAG) extends large language models with external evidence, yet relevant evidence does not ensure that a generator can use it. Indiscriminate retrieval can degrade task performance while adding retrieval and long-context inference costs. Existing methods neither determine before large-scale execution whether a generator, task distribution, and retriever configuration is worth deploying nor diagnose multiple candidate retrievers under a shared budget. We introduce EviCast, a framework that forecasts retrieval value through interventional evidence-state decomposition. EviCast measures a generator's response to controlled evidence states once and combines this shared profile with the offline evidence-state distribution of each candidate retriever, yielding the end-to-end net utility and its 95% lower confidence bound without executing every RAG configuration. A state-refinement analysis characterizes when its binary and topology-aware instantiations are exact and when shared calibration is statistically more efficient than per-retriever calibration. In frozen evaluations, EviCast correctly decides all 14 HotpotQA retrieval configurations with a forecast error of 2.75 F1 points while reducing diagnostic generation calls by 80%, and it attains the lowest forecast error among seven predictors in the low-budget regime. Its topology-aware instantiation, EviCast-T, captures the complementarity between terminal and bridge evidence, correctly deciding all 16 configurations on MuSiQue and HotpotQA and 23 of 24 after prospective transfer to a third dataset. These results show that retrieval deployment depends on both evidence availability and the generator's ability to convert evidence structure into stable net utility. Code is available at https://github.com/YiboWang2002/EviCast.
BibTeX
@misc{wang2026evicast,
  author = {Yibo Wang and Jisheng Dang and Bimei Wang and Jiongpei Hou and Jingwen Zhao and Wenjun Ma and Fei Shen and Juncheng Li and Meng Wang and Qi Tian and Bin Hu and Tat-Seng Chua},
  title = {{EviCast: Forecasting Retrieval Gains Through Shared Response Calibration}},
  year = {2026},
  eprint = {2611.02387},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.02387},
}
Figure