Off-road robots fail most dangerously on terrain that looks fine up close. A patch of smooth mud, the lip of a drop-off, or a shallow puddle on flat ground can match, pixel for pixel, terrain the robot has already driven over safely. Often only the wider scene reveals the danger: water surrounding the patch, sky where ground should continue, a gully line running through it. Current traversability models train on the robot's own driving experience and report high confidence on such patches, because the local evidence matches the positives they were trained on. This proposal makes the model doubt itself in exactly those places. We train two predictors on top of one shared feature extractor. One sees only a small local patch. The other aggregates context from across the whole image. Wherever the two disagree, we apply a confidence penalty, a loss term that rewards spread-out predictions instead of near-certain ones. Everywhere else we leave the model alone.
The core claim is that disagreement between local and long-range predictors is a usable, label-free signal for the specific failure mode the field has named but not solved. Proprioceptive self-supervision, meaning labels derived from the robot's own wheel-slip and motion feedback, only ever confirms terrain after it has been crossed. A uniform confidence penalty cannot single out deceptive pixels. It lowers confidence everywhere, including on clearly safe ground the planner needs to trust. A spatially targeted penalty should cut confident false "traversable" predictions on deceptive terrain without paying that cost elsewhere.
Grounding in Prior Work
Three lines of work frame the problem. The first motivates it. Wild Visual Navigation (Fast Traversability Estimation for Wild Visual Navigation; Wild visual navigation: fast traversability learning via pre-trained models and online self-supervision) learns traversability online from traction feedback, using frozen DINO vision-transformer features pooled into about 100 superpixels. Its authors explicitly name the "cliff problem": traction is excellent right up to the brink. Contrastive Label Disambiguation for Self-Supervised Terrain Traversability Learning in Off-Road Environments projects robot footprints into LiDAR terrain maps, uses a windowed transformer for wider context, and pulls unlabeled patches toward positive prototypes. That design forces confident assignments with no mechanism for abstaining, and it learns only from positives.
The second line supplies the context mechanism. CCNet: Criss-Cross Attention for Semantic Segmentation obtains a full-image receptive field by attending along each pixel's row and column twice in succession, at a fraction of dense attention's memory. GA-Nav uses grouped attention over navigability classes ("safe", "rough", "forbidden") but trains with plain cross-entropy and does not study confidence.
The third line supplies the regularizer, and also constrains it. Regularizing Neural Networks by Penalizing Confident Output Distributions introduced the confidence penalty: subtract the output's entropy, scaled by a weight, from the loss. The same paper reports that its benefit shrinks when other strong regularizers are present. Calibrating Deep Neural Networks using Focal Loss shows that focal loss acts as an implicit entropy regularizer and stays calibrated under distribution shift, where temperature scaling does not. Both apply the penalty uniformly, toward a uniform prior, on whole-image classification. Entropic Graph Regularization in Non-Parametric Semi-Supervised Classification gives the conceptual precedent: entropy terms make predictions in uninformative regions fall back to a safe default.
Our contribution differs from all of these in two ways. The penalty is weighted per pixel by a disagreement signal produced by receptive-field structure. It also pulls toward a terrain prior that leans toward caution, rather than toward a uniform distribution.
Core Mechanism
A frozen DINO backbone produces a grid of patch features, which avoids the superpixel collapse WVN needed for real-time operation. Three heads sit on this grid. The local head is a per-patch multilayer perceptron, so its receptive field is a single patch. The context head runs recurrent criss-cross attention over the patch grid before its own multilayer perceptron, giving it a global receptive field. The fused head takes both representations and produces the prediction the planner uses.
Each head is trained on the available labels. Those are ground-truth navigability groups in the supervised setting, or footprint positives in the self-supervised setting. The fused head additionally receives the penalty
$$\mathcal{L}_{\text{pen}} = \lambda \sum_i \mathrm{sg}\big[w_i\big]\, D_{\mathrm{KL}}\big(p_i \,\|\, \pi\big), \qquad w_i = D_{\mathrm{JS}}\big(p^{\text{loc}}_i \,\|\, p^{\text{ctx}}_i\big).$$
The symbols are as follows. $p_i$ is the fused prediction at patch $i$. $p^{\text{loc}}_i$ and $p^{\text{ctx}}_i$ are the local and context predictions. $D_{\mathrm{JS}}$ is the Jensen-Shannon divergence, a symmetric measure of how different two distributions are. $\mathrm{sg}$ is a stop-gradient, which lets the weight scale the penalty without being optimized itself. $\pi$ is a prior over navigability groups, and $\lambda$ is a scalar weight. With $\pi$ uniform and $w_i \equiv 1$, the term reduces to Pereyra et al.'s confidence penalty, up to a constant.
The stop-gradient matters. Without it, the cheapest way to shrink the penalty is to make the two heads agree, which would destroy the signal. For the same reason we never add a local-context consistency loss. We set $\pi$ to put extra mass on "forbidden", because an uncertain prediction on unverified ground should cost the planner something. At deployment, the planner treats fused entropy as additional cost.
Research Program
The work has three parts. First, we build the three-head model and verify that it runs within a real-time budget on embedded GPU hardware at patch-grid resolution. Second, we construct an evaluation subset that isolates deceptive terrain. Third, we test whether the disagreement weighting, rather than any of its ingredients alone, accounts for the gains.
The deceptive subset is defined independently of the method. On RELLIS-3D (RELLIS-3D Dataset: Data, Benchmarks and Analysis), we generate footprint-positive labels following Xue et al.'s protocol. We then mark as "deceptive" every ground-truth non-traversable pixel whose frozen DINO patch feature has a traversed positive among its nearest neighbors in feature space. The defining property is local similarity to driven terrain, and it is computed with frozen features that no method trains.
We will also report how the disagreement weight $w_i$ is distributed across semantic classes. This tells us whether disagreement concentrates on puddles, mud, water edges, and logs, which are the rare classes RELLIS-3D identifies as catastrophic, or simply on object boundaries. If disagreement is dominated by boundaries, the method is mostly a boundary-softening trick, and we will say so.
Baselines
The naive reference is the same frozen-DINO model with a single fused head trained with plain cross-entropy (or plain positive-unlabeled training in the self-supervised setting), with no penalty and no calibration.
The strongest existing self-supervised method is Xue et al.'s contrastive label disambiguation. On RELLIS-3D it reports pixel accuracy 0.8951 and mIoU 0.6759 (Contrastive Label Disambiguation), against 0.8239 and 0.5382 for a rule-based geometric filter. It is the most direct competitor because it uses footprint labels, wider-than-local context, and the same dataset. We will reimplement it on our footprint labels, report its published numbers alongside, and add RGB-DINO input so that modality does not explain any gap. The file contains no reported calibration error or deceptive-pixel error rate for it, so those numbers will be measured fresh.
For supervised navigability segmentation, the reference is GA-Nav, which reports 5.17–19.06% mIoU improvement on RELLIS-3D without an absolute value in the file. The file also gives HRNet+OCR at 51.55% mIoU on the 20-class RELLIS-3D task. No reported result was found on Lacuna for expected calibration error of any method on RELLIS-3D.
The calibration baselines are the confidence penalty of Pereyra et al. (uniform $w$, uniform prior) and FLSD focal loss from Mukhoti et al. Both are applied to the identical three-head architecture, so the only difference is the weighting. All methods get the same backbone, data split, number of epochs, and a matched hyperparameter search: 20 configurations, tuned on a held-out sequence.
Three ablations isolate the key ingredient. The first replaces $w_i$ with the local head's own entropy, which is a weight based on uncertainty but blind to context. The second replaces $w_i$ with spatially shuffled weights, which preserves how much penalty is applied overall but not where. The third uses the uniform prior instead of the cautious one. We add one more comparison against the obvious design at this intersection: an entropy penalty applied directly to the criss-cross attention maps.
The method beats the baselines if it lowers the false-traversable rate on deceptive pixels at a fixed coverage of safe pixels. Coverage here means the fraction of truly safe pixels the model calls safe with confidence. The comparison is against uniform penalties and focal loss, which must lose no more than one mIoU point overall, and against Xue et al. on pixel accuracy and mIoU.
Why not just…?
Why not just apply temperature scaling after training? A single temperature rescales every pixel identically, so it cannot raise uncertainty on deceptive patches while preserving it on clear ground. Mukhoti et al. also show that temperature-scaled cross-entropy models stay miscalibrated under distribution shift, which is the normal condition when a robot moves between terrains. Temperature scaling on the naive model is included as a baseline.
Why not just use a uniform confidence penalty or focal loss? This is the most serious alternative. On aggregate calibration error it may match or beat us. The proposal stands only if risk-coverage curves on the deceptive subset separate: uniform penalties must lower confidence on safe ground to reach the same error rate on deceptive ground. If they do not separate, the disagreement weighting adds nothing, and we will report that outcome.
Why not just rely on WVN's anomaly head? That head flags feature vectors unlike anything seen in training. Deceptive patches are, by our construction, near traversed positives in feature space, so a feature-novelty detector should miss them. We test this directly by measuring how well a feature-density novelty score from the local features separates errors from correct predictions on the deceptive subset.
Why not just use a deep ensemble? Ensemble members share the same frozen features and the same positive-only labels, so they may agree, confidently and wrongly, on locally familiar patches. They also multiply inference cost on embedded hardware. This is a genuine threat. We run a five-member ensemble at matched training compute and report its deceptive-pixel performance and its latency.
Why not just add criss-cross attention and trust the larger receptive field? If context alone fixed deceptive terrain, the context head by itself would score well. The context-only model is reported as its own row. The RELLIS-3D benchmark results, where large-context models like HRNet+OCR still drop to about 51% mIoU, suggest context alone is not enough.
Decisive Checks
The central experiment is risk-coverage on the deceptive RELLIS-3D subset, comparing four methods: disagreement-weighted, uniform-weighted, focal, and ensemble. The second is the shuffled-weight ablation. If shuffled weights perform as well as true disagreement, the spatial targeting claim fails. The third is a cross-domain check: train on RELLIS-3D, test on RUGD, and measure the false-traversable rate on water and creek pixels, without retuning $\lambda$. The fourth is latency on embedded hardware, compared with WVN's superpixel pipeline.
The method succeeds if all of the following hold. The deceptive-pixel false-traversable rate drops substantially relative to the best uniform penalty. Overall mIoU stays within one point. The shuffled-weight ablation erases the gain.
References
- Fast Traversability Estimation for Wild Visual Navigation
- Wild visual navigation: fast traversability learning via pre-trained models and online self-supervision
- Contrastive Label Disambiguation for Self-Supervised Terrain Traversability Learning in Off-Road Environments
- CCNet: Criss-Cross Attention for Semantic Segmentation
- GA-Nav: Efficient Terrain Segmentation for Robot Navigation in Unstructured Outdoor Environments
- Regularizing Neural Networks by Penalizing Confident Output Distributions
- Calibrating Deep Neural Networks using Focal Loss.
- Entropic Graph Regularization in Non-Parametric Semi-Supervised Classification
- RELLIS-3D Dataset: Data, Benchmarks and Analysis