Some retrieval systems decide on the fly when to search. They watch the language model's uncertainty as it writes and fetch documents only when it seems unsure. Multi-hop questions such as "In what city was the director of Inception born?" are where this matters most. The model must first resolve an intermediate "bridge" entity (the director) before it can search for the final fact. Uncertainty-triggered retrieval has a blind spot here. Models tuned with reinforcement learning from human feedback are often confidently wrong, so a hallucinated bridge entity can come out with near-zero entropy. The trigger never fires, and the rest of the chain builds on a fabricated fact. We propose to put a small learned "thermostat" on the model's hidden states before the trigger is computed. It predicts a temperature for each token, trained specifically on bridge entities in multi-hop chains, and it flattens the next-token distribution exactly where the model is likely to be wrong. The decision about when to retrieve is then made on calibrated uncertainty instead of raw confidence.
Put to a colleague in two sentences: dynamic RAG systems retrieve when the model's entropy is high, but post-RLHF models produce low entropy on wrong bridge entities, so retrieval fails silently. We train a per-token temperature head on whether intermediate hops were actually correct, and use the rescaled entropy as the trigger. We then test whether that beats both simply tuning the trigger threshold and training a plain classifier.
Core Claim
DRAGIN decides when to retrieve with a score that multiplies three things: the raw softmax entropy of each token, the attention that later tokens pay to it, and a semantic-significance weight. It reports 0.3931 F1 on 2WikiMultihopQA with LLaMA2-13B, against 0.3076 for FLARE. The entropy term is uncalibrated. Calibrating Language Models with Adaptive Temperature Scaling (ATS) shows that a single transformer layer on the frozen last hidden state can predict a temperature for every token. Trained with a "selective smoothing" loss (ordinary cross-entropy on correct tokens, a push toward uniform on incorrect ones), it cuts expected calibration error by 10–50% on MMLU, TriviaQA and TruthfulQA at about 7% latency cost.
ATS was built and evaluated only as a scoring tool on single-hop benchmarks. DRAGIN never calibrates. Our claim is that the two should be joined at a specific place. If the temperature head is trained on bridge-entity correctness in multi-hop chains, it will raise the trigger score on confidently wrong intermediate entities. That should lift multi-hop F1 at equal or lower retrieval count than raw-entropy triggering.
Concretely, for token $t$ with logits $z_t$ and hidden state $h_t$, the head outputs $T_\phi(h_t) > 0$. The trigger uses
$$H_t = -\sum_i p_{t,i} \log p_{t,i}, \qquad p_{t,i} = \frac{e^{z_{t,i}/T_\phi(h_t)}}{\sum_j e^{z_{t,j}/T_\phi(h_t)}},$$
in place of raw entropy. The attention-influence and significance terms of DRAGIN stay unchanged. Because the scaling preserves the argmax, greedy generation is untouched. Only the decision to retrieve changes, which makes any gain attributable to the trigger.
One structural point makes or breaks the idea. A single global temperature changes entropy almost monotonically for every token. It is therefore nearly equivalent to moving DRAGIN's threshold and cannot fix a confidently wrong token whose entropy ranks below a correct one. Only a temperature that varies per token can reorder which tokens fire the trigger. The proposal stands or falls on that reordering.
Research Program
The first component is hop-level supervision. MuSiQue builds questions from explicit decompositions with intermediate answers, and 2WikiMultiHopQA comes with explicit reasoning paths. We run the frozen backbone (LLaMA2-13B, to match DRAGIN, plus one newer aligned model) in closed-book chain-of-thought mode on training questions. We align generated spans to gold intermediate entities by normalized string match, and label each bridge-entity token correct or incorrect.
The second component is the temperature head. It uses the ATS architecture and selective smoothing loss, trained on these labels with the backbone frozen. Ordinary tokens are trained under the standard ATS objective with smaller weight, so that calibration on connective text does not drift.
The third component plugs the head into DRAGIN's information-need detector, with DRAGIN's attention-based query formulation unchanged. We then sweep the trigger threshold to trace F1 against average retrievals per question.
We also run a diagnostic analysis that does not depend on end-to-end QA. On held-out chains, we measure how well each trigger score separates wrong from right bridge entities (area under the ROC curve), and the calibration error restricted to bridge tokens. If the head does not improve detection of wrong bridge entities, any end-to-end gain must come from somewhere else, and we will report it that way.
Baselines
The naive reference points are no retrieval and single-shot retrieval on the original question. They bound what any trigger can add. The main existing method is DRAGIN with raw entropy, run with the authors' configuration and LLaMA2-13B. Its reported 0.3931 F1 on 2WikiMultihopQA (DRAGIN) is the number our reproduction must first match before any comparison counts. DRAGIN is the strongest competitor on this question because it is the closest system that already uses token uncertainty to decide when to retrieve; only its uncertainty signal differs from ours. Beating it means higher F1 at matched average retrieval count, or equal F1 with fewer retrievals, across the whole threshold sweep rather than at one hand-picked point.
The second baseline is DRAGIN with a global temperature tuned jointly with the threshold on the same validation split and the same tuning budget as our head. This isolates whether per-token variation matters at all. The third is an off-the-shelf ATS head trained on its original single-hop data and plugged into the trigger. This isolates whether training on bridge entities matters, as opposed to generic calibration. The fourth is a direct probe: a classifier on the same hidden states, same labels and same capacity, that predicts "bridge entity wrong" and fires retrieval when its probability crosses a threshold.
For context, we also report an always-retrieve interleaved loop in the style of IRCoT. It gives a high-retrieval, high-latency reference. Stronger planning systems such as REAP, which reports 79.6% F1 on 2WikiMultiHopQA and 38.3% on MuSiQue, use different backbones and pipelines. We cite them only as context, not as head-to-head competitors.
Why not just…?
Why not just tune DRAGIN's threshold, or a global temperature? A global temperature preserves, to first order, the ordering of tokens by entropy. Threshold tuning can therefore trade recall for retrieval count but cannot catch a wrong bridge entity that is more confident than a correct one. ATS's reported 10–50% calibration gains over global scaling (ATS) suggest that miscalibration is token-dependent. Baseline two tests this directly: if the jointly tuned global temperature matches our curve, the claim fails.
Why not just train a classifier on hidden states to predict when to retrieve, and skip temperature entirely? This is the real threat, and we treat it as one. With the same labels and capacity, a free-form probe may match the temperature head in-distribution. Our hypothesis is that constraining the signal to rescale the model's own distribution generalizes better across datasets. The probe can latch onto dataset-specific surface cues, whereas the temperature head can only modulate an entropy the model already produces. The scaled distribution also gives a calibrated answer confidence for free. The decisive test is cross-dataset: train both on MuSiQue, then test on 2WikiMultiHopQA and HotpotQA. If the probe matches there too, we will report that the temperature form is unnecessary and that the contribution reduces to hop-level supervision.
Why not just sample several chains and use disagreement, or use distractor sensitivity as in CaliDist? Self-consistency needs 15–40 passes, and CaliDist needs 3–5 (CaliDist). Both act at the answer level, after the chain is written, not at the moment a bridge entity is generated. Revisiting Self-Consistency adapts sampling temperature from vote margins, but only for closed-book math. We include a five-sample bridge-entity disagreement trigger at matched compute as an extra comparison. It remains the right choice for black-box APIs, where our white-box head cannot run.
Proposed Experiments
We evaluate on 2WikiMultiHopQA, HotpotQA and MuSiQue, with StrategyQA as an implicit-reasoning check. We report exact match and F1, average retrievals per question, wall-clock latency, and the trigger diagnostics: separation of wrong from right bridge entities, and calibration error on bridge tokens.
The first decisive check is the per-token reordering. Among bridge entities that were wrong and had raw entropy below DRAGIN's threshold, what fraction does the calibrated trigger now fire on, and what fraction of correct low-entropy tokens does it falsely flag? The second is the F1-versus-retrievals curve against all baselines. The third is the cross-dataset probe comparison described above. The fourth is an ablation that drops the bridge-entity weighting from training, to show whether the hop-specific supervision is what does the work.
The outcome that would refute the proposal is simple to state: a global temperature or a plain probe reproduces our curve. In that case the field learns that calibration as such is not the missing piece in uncertainty-triggered multi-hop retrieval.
References
- DRAGIN: Dynamic Retrieval Augmented Generation based on the Real-time Information Needs of Large Language Models
- Calibrating Language Models with Adaptive Temperature Scaling
- Interleaving Retrieval with Chain-of-Thought Reasoning for Knowledge-Intensive Multi-Step Questions
- REAP: Enhancing RAG with Recursive Evaluation and Adaptive Planning for Multi-Hop Question Answering
- CaliDist: Calibrating Large Language Models via Behavioral Robustness to Distraction
- Revisiting Self-Consistency from Dynamic Distributional Alignment Perspective on Answer Aggregation