Some robot navigators built on vision-language models (VLMs, models that answer questions about images in text) "imagine" what lies around a corner. At each step they use a diffusion model, an image generator that gradually denoises random noise into a picture, to render plausible views of unexplored directions. The VLM then picks a direction by comparing these imagined views. This helps in unfamiliar buildings, but it is expensive: one such system reports about 4.2 seconds of A100 time per generated image. It is also risky, because generated views contain artifacts that mislead the planner. Today these systems imagine at every step, or switch imagination on with hand-tuned similarity rules. Nothing tells the agent whether this particular image will change its decision for the better.
We propose to learn that signal directly. A small probe reads the frozen VLM's hidden states and predicts the gain from imagining: the probability that the decision made with imagined views makes more progress toward the goal than the decision made without them. The probe is trained on outcomes the simulator labels for free (geodesic distance to the goal in training scenes). A conformal threshold then bounds how often the agent skips imagination when imagination would have helped. The two-sentence version is this. Instead of asking "is the model unsure?", we ask "will spending a diffusion call here change the outcome for the better?", and we answer with a calibrated probe rather than the model's own overconfident output. If that works, an agent can keep most of the success of always-on imagination at a fraction of the generation cost, and it can refuse imagination in situations where generated views tend to hallucinate.
Core Claim
A probe on frozen VLM hidden states, trained to predict the counterfactual benefit of generative imagination, gives a better compute-versus-success trade-off than four alternatives: always imagining, uncertainty-triggered imagining (entropy or verbal confidence), and similarity-heuristic scheduling. "Better" means higher success at matched numbers of diffusion calls on unseen HM3D and HSSD scenes. The offline cost of producing the probe's training labels is recovered within a measurable number of deployment episodes.
The distinction from uncertainty gating is the crux. An agent can be uncertain in places where imagination cannot help, such as a dead-end room or a view the generator renders badly. It can also be confident and wrong in places where a single imagined view would reveal a doorway. Overconfidence makes the second case common. Liu et al.'s PROBECAL work (Hao Liu et al.) shows that output-token confidence in multi-step agent traces is badly miscalibrated: expected calibration error (ECE, the average gap between stated confidence and actual accuracy) is 0.690 on TabMWP. A probe on hidden states brings it to 0.035. We reuse that technique, a small MLP on frozen embeddings, but point it at a different target.
Positioning Against Prior Work
The closest work is VISTA (VISTA). It generates goal images with a LoRA-tuned Stable Diffusion model. Its Adaptive Imagination Scheduler toggles between static and dynamic imagination using trajectory uncertainty and visual-semantic similarity. That scheduler is a heuristic and is not calibrated. It also decides which kind of imagination to run rather than whether to pay for any at a given step, and VISTA is a trained system rather than a zero-shot one. We keep the scheduling idea but replace the heuristic with a learned, calibrated estimate of decision gain, inside a zero-shot planner.
DreamPhase (DreamPhase) supplies the motivating precedent of a safety gate on imagination. It penalizes imagined branches by Monte-Carlo-dropout uncertainty and falls back to real interaction when uncertainty is high. Its imagination is a cheap latent model (about 9 ms per rollout) in discrete text and grid domains such as BabyAI and MAZE. Pixel-level imagination in photorealistic 3D is where the cost and the hallucination problem actually bite, and that is our setting.
Explore until Confident (Explore until Confident) motivates and supplies the statistical tool. It shows VLM softmax scores cause premature stopping in embodied question answering and uses multi-step conformal prediction for a coverage guarantee on when to stop. It calibrates only the final answer. We apply conformal risk control to an intermediate, per-step resource decision.
ImagineNav (ImagineNav) is our testbed. It predicts candidate waypoints with a ResNet-18, renders views with the PolyOculus diffusion model, and asks GPT-4o-mini to choose. It is mapless and zero-shot, and it acknowledges that synthesis artifacts mislead the VLM.
Hierarchical zero-shot planners that self-correct form the constraining context. Three-Step Nav (Three-Step Nav) audits its own trajectory through verbal prompts. MerNav (MerNav) triggers recovery with fixed rules. Both show that agents benefit from knowing when they are going wrong. Kapoor et al. (Large Language Models Must Be Taught to Know What They Don't Know) show that verbalized confidence swings with prompt wording and is unreliable. The same paper offers a useful data point for us: calibration tuning on roughly 1,000 graded examples cut open-ended MMLU ECE from about 30% to 10%. That suggests our probe will not need huge label sets.
Research Program
The system has a frozen open-weight VLM, a waypoint proposer and diffusion view generator as in ImagineNav, and the gain probe. We must use an open VLM such as Qwen2-VL (the captioner family VLM used in VISTA) rather than GPT-4o-mini, because probing needs hidden states. That substitution may lower absolute success. We will therefore report the ungated open-VLM agent and, where budget permits, the original GPT-4o-mini pipeline, so the reader can separate gating effects from backbone effects.
To build labels, we run the agent in HM3D and HSSD training scenes and fork each step into two branches. One branch chooses a waypoint from real observations only; the other also gets imagined views. The simulator gives each branch's geodesic progress toward the goal. The training target is whether the imagined branch makes strictly more progress. A second head predicts whether the no-imagination choice makes progress at all, which gives a calibrated correctness estimate that can be compared with uncertainty baselines. Probe inputs are VLM hidden states computed before any generation, so querying the probe costs one forward pass the agent runs anyway.
At deployment, the agent calls the generator only when the predicted gain exceeds a threshold. The threshold is set by conformal risk control on a held-out calibration split of training scenes, so the rate of "skipped imagination that would have helped" stays below a user-chosen level. An optional third head reads features of the imagined views after generation and predicts whether the post-imagination choice is worse than the pre-imagination one. This targets the hallucination failure that ImagineNav reports.
This design makes the synthetic-overhead question explicit. Every training label costs one diffusion call plus two simulator rollouts. We report that offline GPU-hour cost and compute the break-even number of deployment episodes after which online savings exceed it. We also measure how success varies with label count (500, 1,000, 5,000, 20,000 steps) to test whether the small-data plateau Kapoor et al. observed carries over. The motivation is the evidence from HSSD-200 (HSSD-200) and DISCO (DISCO) that small, high-quality synthetic data beats large noisy data. Our version of that argument applies to online synthesis: fewer, better-placed generations.
Baselines
The naive reference points are two ungated agents run with the same frozen VLM and proposer. The first never imagines and sets the floor. The second always imagines (ImagineNav-style) and sets the success ceiling that we try to approach at lower cost. On published numbers, ImagineNav reports success-rate gains of 4.0% on HM3D and 23.5% on HSSD over prior mapless methods (ImagineNav). We found no absolute ImagineNav success rate on Lacuna, so we will reproduce both reference agents ourselves on identical episodes. For context on open-vocabulary object navigation, MerNav reports 45.7% success on HM3D_OVON (MerNav); we will include that benchmark if our pipeline supports free-form goals. Random gating at matched call rates is the trivial control that any gate must beat.
The strongest competitor on the specific question of when to imagine is a reimplementation of VISTA's Adaptive Imagination Scheduler logic inside our pipeline. It triggers imagination from trajectory uncertainty and visual-semantic similarity, with its thresholds tuned on the same calibration split and the same tuning budget as our conformal threshold. VISTA reports 74.9% success on R2R unseen environments (VISTA). That number comes from a trained system on discrete R2R, so it is not directly comparable to our zero-shot continuous setting, and we will not claim to beat it in absolute terms.
The strongest uncertainty competitor is entropy gating: the entropy of the VLM's distribution over candidate waypoints, after temperature scaling fitted on the calibration split. Verbal-confidence gating, where the VLM rates its confidence and imagination triggers below a threshold, matches the prompt-based audit style of Three-Step Nav. Each gate is swept over thresholds to trace a full success-versus-calls curve, and comparisons are made at matched average diffusion calls per episode.
Two ablations isolate our key ingredient. The first trains the same probe architecture on the correctness target instead of the gain target, which separates "learned uncertainty" from "learned value of imagination." The second replaces simulator geodesic labels with labels from a VLM grader, which tests whether free geometric supervision avoids the teacher-noise problem Kapoor et al. flag.
Beating the strongest baselines means two things. First, success at 30% of always-imagine's diffusion calls should be statistically indistinguishable from always-imagine's success, while the AIS-style and entropy gates fall significantly short at the same budget. Second, the area under the success-versus-calls curve should be higher than every baseline's, with 95% bootstrap intervals over episodes that exclude overlap.
Why not just…?
Why not just gate on temperature-scaled entropy of the VLM's choice? This is the most serious threat. Temperature scaling fixes calibration but leaves the ranking of steps unchanged, and gating depends only on ranking. Entropy gating therefore succeeds exactly to the extent that "uncertain" coincides with "imagination helps." Liu et al. (Hao Liu et al.) and Explore until Confident (Explore until Confident) both find that output probabilities in agent settings are confidently wrong, which predicts that entropy will miss high-confidence errors. That is evidence from adjacent tasks, not ours. The correctness-target ablation settles the question. If entropy gating matches the gain probe's curve, our central claim fails, and the honest conclusion is that a black-box signal suffices.
Why not just ask the VLM whether it is sure, as Three-Step Nav does? It requires no training and works with closed models. Kapoor et al. (Large Language Models Must Be Taught to Know What They Don't Know) show that verbalized confidence shifts with prompt phrasing without tracking accuracy. We include it as a baseline, report its ECE on per-step correctness, and measure its sensitivity across three paraphrased prompts.
Why not just imagine every k steps, or use a cheaper generator? A fixed schedule is a matched-budget control, and we include it beside random gating. A faster generator reduces the cost of each call but not the hallucination problem, and it is complementary: any gate multiplies whatever savings a cheaper generator provides. If a fixed schedule matches learned gating, then imagination's value is roughly uniform across steps, and we will report that outcome.
Why not just replace pixels with a latent world model, as DreamPhase does? That would remove diffusion latency, but DreamPhase needs logs from the target environment and has been shown only in discrete domains with horizons under 100 steps (DreamPhase). Building a photorealistic latent simulator is a separate project. Our gate would apply equally to latent imagination if one existed.
Proposed Experiments
The primary study is zero-shot object-goal navigation on HM3D and HSSD validation scenes, with the probe trained only on training scenes. We report success rate, success weighted by path length (SPL), diffusion calls per episode, and wall-clock seconds per episode. The ImagineNav setup sets the protocol for these benchmarks.
For the vision-language navigation transfer study, the same gate is attached to an imagination-augmented zero-shot planner on R2R-CE val-unseen. Three-Step Nav reports SR 34%, NE 5.87, and SPL 29.12% on 100 sampled episodes (Three-Step Nav). We will evaluate on the full split, plus that 100-episode subset for comparability.
Calibration quality is reported directly. We give ECE and Brier score for the correctness head across every gating signal, and AUROC for the gain head in predicting realized gain. A cross-dataset shift test trains the probe on HSSD and tests on HM3D, which asks whether calibration survives the synthetic-to-real-scan gap that HSSD-200 found matters for success (HSSD-200). We also check whether the empirical rate of harmful skips stays within the conformal target α under that shift.
Three decisive checks determine whether the idea holds. First, at matched diffusion budgets, the gain probe must beat temperature-scaled entropy; if not, the counterfactual-gain target adds nothing. Second, the post-generation hallucination head must lower the rate at which imagination makes decisions worse, compared with always trusting imagined views. Third, the break-even analysis must show that the probe's offline labeling cost is repaid within a deployment horizon of a few hundred episodes. If labeling costs more than it saves, then for this regime the overhead problem has moved offline rather than been solved.
References
- Uncertainty Calibration for Tool-Using Language Agents
- VISTA: Generative Visual Imagination for Vision-and-Language Navigation
- DreamPhase: Offline Imagination and Uncertainty-Guided Planning for Large-Language-Model Agents
- Explore until Confident: Efficient Exploration for Embodied Question Answering
- ImagineNav: Prompting Vision-Language Models as Embodied Navigator through Scene Imagination
- Three-Step Nav: A Hierarchical Global–Local Planner for Zero-Shot Vision-and-Language Navigation
- MerNav: A Highly Generalizable Memory-Execute-Review Framework for Zero-Shot Object Goal Navigation
- Large Language Models Must Be Taught to Know What They Don’t Know
- Habitat Synthetic Scenes Dataset (HSSD-200): An Analysis of 3D Scene Scale and Realism Tradeoffs for ObjectGoal Navigation
- DISCO: Distilling Counterfactuals with Large Language Models