Legged and wheeled-legged robots are starting to ask vision-language models (VLMs, large models that take an image and answer questions about it in text) what the ground ahead is like. Examples include "is that a ramp or a wall?", "is this floor slippery?", and "what is the friction coefficient?" The answers go straight into the controller. The trouble is that these answers are summaries of a scene, and they fail the way text summaries fail. They can state things the image does not support, or contradict evidence the robot already has. A summarization researcher would call this abstractive hallucination: fluent output that is not grounded in its source. A robot that believes a hallucinated "dry stone" walks onto wet tile at full speed.
We propose to treat a VLM's terrain description the way factual-consistency research treats a generated summary. We split it into small, checkable claims. Each claim is tested against the evidence the robot actually owns. Claims about shape are checked against the elevation map. Claims about material, such as slipperiness or softness, are checked against slip and contact-force history from ground the robot has already crossed. Any claim the robot has no way to check is labelled "unverifiable" rather than trusted. Only verified claims change the controller. Contradicted or unverifiable claims fall back to conservative defaults. In two sentences: the VLM proposes, the robot's own sensors act as the source document, and each sentence of the proposal must be backed by that document before it can make the robot faster or bolder.
Core Claim
The claim is that per-claim, evidence-typed verification with an explicit "unverifiable" outcome removes most of the harm VLM terrain hallucinations cause, while keeping most of the benefit of acting before a slip happens. We further claim that it beats both whole-answer self-checking and purely reactive correction. The idea transfers the most useful lesson of SummaC. That paper made inconsistency detection work by matching granularity: it scored each summary sentence against each source sentence instead of scoring a whole summary against a whole document. Here the "source sentences" are map patches and slip episodes, not text.
The summarization split between intrinsic and extrinsic hallucination maps directly onto the robot setting. An intrinsic terrain hallucination contradicts present evidence, as in "the path is flat" when the height map shows a 20 cm step. An extrinsic one asserts a property no current sensor can confirm, as in "low friction, about 0.3" inferred from appearance alone. The two need different treatment. Intrinsic claims should be rejected. Extrinsic claims should be held in a pending state until the robot gathers evidence, for example by probing cautiously with a foot or wheel.
Relation to Prior Work
Three robotics papers motivate the gap. Friction-Aware Safety Locomotion for Wheeled-legged Robots has GPT-4o, aided by CLIP image retrieval, predict friction from a camera frame and feeds that number straight into a reinforcement-learning policy. It reports a friction RMSE of about 0.25 and 17/17 hardware successes, but it never checks whether a given prediction is grounded. Our verifier sits exactly between its friction module and its policy. SARO uses LLaVA-34B to break terrain crossing into sub-tasks. It adds a VLM discriminator that judges from images whether each sub-task is finished, and that discriminator matters: ramp success falls from 67% to 10% without it. However, SARO checks the VLM with the VLM itself, in binary form, from 2D frames. We check against metric 3D and physical evidence, one claim at a time. Coupling Vision and Proprioception for Navigation of Legged Robots shows that vision and felt slip disagree, and it re-weights cost maps when they do. That is the reactive half of our design. It has no notion of which visual assertion was wrong, and no way to refuse an unverifiable one before stepping.
Two papers supply method. SummaC supplies the pairwise score matrix with max-then-aggregate scoring. Rejecting Hallucinated State Targets during Planning supplies the split between permanently infeasible and temporarily infeasible hallucinated targets, plus the practice of training a feasibility checker on deliberately generated bad targets. That work stays inside discrete grid worlds without language. We carry both ideas into continuous terrain with natural-language claims.
One paper constrains the design. Improving Factual Consistency of News Summarization by Contrastive Preference Optimization finds that modern models make sophisticated, coherent fabrications rather than simple entity swaps, and that perturbation-based training misses them. So our "hallucinated claim" training data cannot be only random number swaps. It must include the VLM's own natural errors.
Research Program
The system has three parts. The first is a claim extractor. The VLM is prompted to answer in a fixed schema of atomic assertions, each tied to a region of the image. Examples are a step height in centimetres, the continuity of a surface, the presence of a gap, a material label, and a friction value. Each region is projected into the robot-centric elevation map using the depth data. Forcing this structure is cheap and removes the sentence-fusion problem that SummaC itself cannot handle.
The second part is the evidence base. Geometric evidence comes from elevation_mapping_cupy, which already runs at about 50 Hz on embedded GPUs. Physical evidence is a short-term memory of slip episodes: foot or wheel slip, impact profiles, and contact forces. Each episode is stored with the location where it happened and the appearance of that patch.
The third part is the verifier. For a claim $c$ and evidence items $e_1,\dots,e_M$, a small learned model outputs probabilities of support, contradiction, and no relevant evidence for each pair. Following SummaC's zero-shot rule, the claim score is
$$s(c) = \max_j \big[p_{\text{support}}(c, e_j)\big] - \max_j \big[p_{\text{contra}}(c, e_j)\big],$$
and claims whose evidence is all "no relevant evidence" are labelled unverifiable.
Geometric checks can be partly analytic, such as comparing a claimed step height with measured patch statistics. Material checks are learned. They ask whether patches that look like this one produced slip consistent with the claimed friction. The controller then uses verified values and replaces the rest with conservative priors, such as low assumed friction and reduced speed. This keeps the policy from the friction-aware work unchanged, so the comparison is clean.
The verifier is trained on two kinds of labelled pairs. The first comes from logged runs on GrandTour, whose camera, depth, LiDAR, IMU, and foot-contact streams give both geometric and slip ground truth. The second is a simulated terrain suite with appearance–physics mismatches. Following the "generate bad targets" lesson, it pairs the VLM's own wrong claims with hard synthetic contradictions.
Baselines
The naive reference is the open-loop pipeline of Friction-Aware Safety Locomotion. VLM outputs go to the policy unchecked, and the paper reports a friction RMSE of about 0.25 and 17/17 trials on its five surfaces. We re-run it with the same VLM, prompts, and policy as our system, so the only difference is the verifier. Beating it means fewer slip and tip events on mismatch terrains, with traversal time no more than modestly higher.
The strongest existing self-checking method is SARO's VLM discriminator, generalized so that the same VLM judges its own terrain claims from the image. Its reported numbers are 85.31% simulation success, against 49.30% for Rapid Motor Adaptation, and the 67%-versus-10% ramp ablation. We run it on our terrain suite with the same VLM and the same per-step call budget. To beat it, our verifier must achieve higher claim-level balanced accuracy on hallucination detection and fewer falls at equal or lower latency.
The reactive baseline is VP-Nav-style re-weighting of the cost map after felt slip, with no VLM gating. It gets the same proprioceptive streams. A dedicated small vision network that regresses friction with its own uncertainty is trained on the same labelled data and the same tuning budget as our verifier. Two ablations isolate the key ingredients. The first scores whole answers instead of individual claims. The second removes the "unverifiable" class, which forces support or contradiction. We also report SummaC applied off the shelf to text captions of the map as a text-only reference point. Its 74.4% balanced accuracy on the SummaC benchmark gives a rough ceiling for what text NLI achieves in its home domain. No reported result was found on Lacuna for any detector of VLM terrain hallucination.
Why not just…?
Why not just correct after slipping, as VP-Nav does? Reactive correction is free of hallucination but pays for every error with a slip. On wheeled inverted pendulums, purely reactive control failed outright on wood in the friction-aware study. Our verifier uses the same slip evidence, but also uses it ahead of time on patches that look similar. The reactive baseline measures how much of our gain comes from acting early rather than from verification. If it matches us on falls, then the proactive half of the design is not worth having.
Why not just ask the VLM to check itself, or sample several answers and measure agreement? Self-checks draw on the same image and the same prior beliefs about what surfaces look like, so they fail together. Glossy dry floors and matte wet ones fool every sample the same way. SARO's discriminator helps for judging whether a sub-task is done, but it has no metric or physical evidence. We test this directly by measuring how often the self-check and our verifier miss the same claims on appearance-mismatch terrain.
Why not just drop language and train a small friction regressor with calibrated uncertainty? This is a real threat. For a single scalar like friction, it may be as good as our system and cheaper. The language layer earns its place only if verification pays off on mixed, structured claims, meaning geometry, material, and affordance together, or for VLMs used open-vocabulary in new scenes. The verifier is also model-agnostic, so we will run it on the regressor's outputs as well. If the regressor alone matches the full system on friction and on combined tasks, we will report that the contribution is the evidence-typed verifier, not the VLM.
Proposed Experiments
The first experiment is offline. On GrandTour sequences, we collect VLM claims and label each one as supported, contradicted, or unverifiable using elevation and contact-based slip ground truth. We report claim-level balanced accuracy for every detector, split into intrinsic and extrinsic claims and by condition, including snow, tunnels, and dust, where GrandTour documents perception degradation.
The second experiment is in simulation. In an IsaacGym terrain suite with deliberate appearance–physics mismatches and with SARO-style ramps next to walls that look like ramps, we measure success, slip and tip rate, and time to goal for all baselines. The decisive check is the mismatch subset. Our system should cut falls relative to the open-loop pipeline and the self-check, while staying close to open-loop speed on terrain where appearance and physics agree. If it only wins by being uniformly slower, the claim fails.
The third experiment runs on hardware: a wheeled-legged or quadruped platform repeating the five-surface protocol from the friction-aware study, plus wet and polished variants. We report trial success counts alongside verifier latency, which must stay below the control replanning period.
References
- SummaC: Re-Visiting NLI-based Models for Inconsistency Detection in Summarization
- Friction-Aware Safety Locomotion for Wheeled-legged Robots using Vision Language Models and Reinforcement Learning
- SARO: Space-Aware Robot System for Terrain Crossing via Vision-Language Model
- Coupling Vision and Proprioception for Navigation of Legged Robots
- Rejecting Hallucinated State Targets during Planning
- Improving Factual Consistency of News Summarization by Contrastive Preference Optimization
- Elevation Mapping for Locomotion and Navigation using GPU
- GrandTour: A Legged Robotics Dataset in the Wild for Multi-Modal Perception and State Estimation