VoxSAOR: Semantic-Aware Occupancy Robustification via Aleatoric Uncertainty Modeling

Mengfei Duan, Yuheng Zhang, Guoqiang Zhao, Yuanfan Zheng, Kailun Yang, Zhiyong LiarXiv 2026 · arXiv

TL;DR

VoxSAOR is a new framework that makes 3D occupancy prediction more robust by explicitly modeling aleatoric uncertainty—the inherent "fuzziness" caused by occlusions and depth errors. By training an ensemble of prediction heads with reliability-aware feature perturbations, it achieves better accuracy and more trustworthy confidence scores without adding any computational overhead during inference.

What Problem Does This Paper Address?

In autonomous driving and robotics, "3D occupancy prediction" is the task of turning 2D camera images into a dense 3D grid of voxels, where each voxel is labeled (e.g., "road," "car," or "empty").

The problem is that current models usually assume the features extracted for these voxels are perfectly reliable. In the real world, they aren't. Occlusions (one car blocking another), depth ambiguity (is that object 10 meters away or 12?), and projection errors create spatially varying reliability. Some voxels are "stable," while others are "ambiguous."

Most existing methods ignore this variation, leading to brittle predictions, especially at object boundaries or in occluded regions. Furthermore, these models are often "overconfident" in their mistakes, which is a major safety concern for self-driving systems.

Methods

The authors propose VoxSAOR (Semantic-Aware Occupancy Robustification), which introduces two main components:

1. Semantic-Aware Aleatoric Uncertainty Modeling (SAUM)

Since we don't have ground-truth labels for "uncertainty," the authors create a proxy.

  • Local Semantic Consistency (LSC): They measure how much a voxel's label agrees with its neighbors. High agreement suggests a stable region; low agreement suggests a boundary or an ambiguous area.
  • Cross-View Semantic Consensus (CSC): They use 2D pseudo-labels (from a Vision-Language Model like OpenSeeD) to ensure the 3D voxel features actually align with what the 2D camera sees.
  • Feature Transitions: They match "stable" voxels with "ambiguous" ones of the same class to model how features shift when they become unreliable. This is captured as a class-conditional Gaussian distribution.

2. Aleatoric Regime Ensemble Learning (AREL)

Instead of training one prediction head, they train an ensemble (typically $R=2$).

  • During training, they inject noise (perturbations) into the voxel features. The "strength" and "direction" of this noise are guided by the Gaussian proxies learned in the SAUM step.
  • Each head is exposed to a different "regime" of noise, forcing the model to learn semantic consistency even when the input features are degraded.
  • At Inference: The noise-making and proxy-modeling are turned off. All heads process the same "clean" features, and their outputs are averaged. This provides a "deterministic consensus" that is more robust than a single head, with zero extra cost in terms of sampling or latency.

Datasets

The authors evaluated VoxSAOR across three major benchmarks:

  • SemanticKITTI: The gold standard for outdoor autonomous driving scenes.
  • SSCBench-KITTI-360: A large-scale driving dataset with diverse urban sequences.
  • Human360Occ (H3O): A panoramic 3D occupancy dataset designed for "embodied perception" (e.g., robots or humans moving through a simulated environment).

Key Insights and Ideas

  • Uncertainty as a Training Signal: The core "aha" moment is that you don't need to predict uncertainty at runtime to benefit from it. By using uncertainty to "stress-test" the model during training, you build a more robust internal representation.
  • Reliability-Aware Perturbation: Random noise (like Gaussian blur) isn't enough. By using "stable-to-ambiguous" transitions, the noise actually mimics the specific ways that depth errors and occlusions corrupt features in 3D space.
  • Architecture Agnostic: VoxSAOR isn't a new backbone; it's a "wrapper." The authors proved it works on top of several popular architectures like VoxDet, CGFormer, and MonoScene.

Key Results

  • Accuracy Boost: On SemanticKITTI, integrating VoxSAOR into the VoxDet backbone improved the Mean Intersection over Union (mIoU) from 18.23% to 18.64% and the scene completion IoU from 46.88% to 48.02%.
  • Long-Tail Recognition: It significantly improved performance on rare classes (the "long tail"). For example, it saw a +0.62 percentage point gain in tail mIoU on the SemanticKITTI validation set.
  • Better Calibration: The model became less "arrogant." It showed lower Expected Calibration Error (ECE) and Negative Log-Likelihood (NLL), meaning when the model says it's 90% sure, it's actually right about 90% of the time.
  • Qualitative Improvement: Visualizations show that VoxSAOR produces much cleaner object boundaries and more coherent scene layouts compared to baselines, which often produce "scattered" or "bleeding" voxel labels.

Limitations and Future Work

  • Offline Pseudo-labels: The current training pipeline requires 2D pseudo-labels generated offline by a separate model (OpenSeeD). Integrating this into an end-to-end online process would be more elegant.
  • Static Focus: The current work focuses on static 3D snapshots. The authors suggest that extending this to spatiotemporal (4D) representations—modeling how uncertainty evolves over time—is a promising next step.
  • Downstream Integration: While the occupancy maps are better, the paper does not explicitly measure how much this improves actual downstream tasks like path planning or collision avoidance in a closed-loop system.

For more details, the source code is available at https://github.com/MengfeiD/VoxSAOR.

Summary updated

Abstract

Vision-based 3D occupancy prediction provides a dense semantic representation of scenes for autonomous driving and embodied perception, yet remains vulnerable to occlusion, depth ambiguity, and projection errors that induce spatially varying reliability in voxel features. Existing methods typically map voxel representations to semantic predictions without explicitly modeling such observation-induced feature variations. To address this limitation, we propose VoxSAOR, a semantic-aware occupancy robustification framework that models aleatoric uncertainty through feature-space proxies. It first quantifies Local Semantic Consistency (LSC) to derive a voxel-wise structural stability prior and uses cross-view semantic consensus to filter unreliable semantic correspondences. Within each semantic category, spatially matched stable and ambiguous voxels form stable-to-ambiguous feature transitions, which are modeled by a class-conditional diagonal Gaussian distribution, capturing both feature displacement and dispersion. Guided by these proxy distributions, Aleatoric Regime Ensemble Learning (AREL) trains independently parameterized prediction heads using feature perturbations at progressively increasing scales, with perturbation strength modulated by structural reliability. At inference, proxy modeling and sampling are disabled, and all heads process the same observed features; their output probabilities are then averaged to produce a deterministic consensus prediction without requiring sampling. Experiments across multiple datasets and occupancy backbones demonstrate the effectiveness of VoxSAOR in improving semantic occupancy prediction, long-tail recognition, and predictive calibration. The source code will be made publicly available at https://github.com/MengfeiD/VoxSAOR.
BibTeX
@misc{duan2026voxsaor,
  author = {Mengfei Duan and Yuheng Zhang and Guoqiang Zhao and Yuanfan Zheng and Kailun Yang and Zhiyong Li},
  title = {{VoxSAOR: Semantic-Aware Occupancy Robustification via Aleatoric Uncertainty Modeling}},
  year = {2026},
  eprint = {2611.20507},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.20507},
}
Figure