VisHall3D: Monocular Semantic Scene Completion from Reconstructing the Visible Regions to Hallucinating the Invisible Regions

Haoang Lu, Yuanqi Su, Xiaoning Zhang, Longjun Gao, Yu Xue, Le WangICCV 2025

TL;DR

This paper introduces VisHall3D, a novel two-stage approach for monocular semantic scene completion. It tackles the challenge of reconstructing 3D environments from a single 2D image by explicitly separating the task into reconstructing visible areas and "hallucinating" occluded or out-of-view regions, leading to significantly improved accuracy and geometric consistency.

What Problem Does This Paper Address?

Autonomous vehicles and robots need to understand their surroundings in 3D to navigate safely and effectively. Traditionally, this has relied on expensive sensors like LiDAR. Monocular Semantic Scene Completion (Monocular SSC) aims to achieve this using only a single RGB camera, making it a more cost-effective and flexible solution. However, existing monocular SSC methods struggle with two main issues:

  1. Feature Entanglement: Features learned for visible parts of the scene get mixed with those for invisible parts, leading to confusion.
  2. Geometric Inconsistency: The reconstructed visible and invisible regions don't align well or have conflicting structures.

These problems arise because a single image inherently provides incomplete information. Some parts of the scene are directly visible, while others are hidden behind objects (occluded) or are simply outside the camera's field of view (out-of-view). Accurately reconstructing the entire 3D scene requires not only understanding what's visible but also intelligently inferring what's not.

Methods

VisHall3D proposes a two-stage framework to address these challenges:

  1. VisFrontierNet (Visible Region Reconstruction): This module focuses on accurately reconstructing the visible parts of the scene. It introduces a "visibility-aware projection module" that precisely traces the "visual frontier" – the boundary between visible and occluded/out-of-view regions. This is achieved by:
  2. Unsigned Distance Function: Instead of just using binary visibility, it calculates the unsigned distance of voxels to the visual frontier. This provides a richer geometric prior, helping to better capture the scene's geometry.
  3. Feature Refinement: It uses 3D deformable attention to dynamically adjust its focus on the most informative parts of the visual frontier, leading to more detailed reconstructions.
  4. OcclusionMAE (Invisible Region Hallucination): This module takes the output from VisFrontierNet (specifically, the noisy predictions for visible voxels) and generates plausible geometries for the occluded and out-of-view regions. It employs a denoising Masked Autoencoder (MAE) approach:
  5. Noise Injection: It adds controlled noise to the visible voxel predictions. This is crucial because the network needs to learn to recover from imperfect inputs.
  6. Adaptive Batch Normalization: The level of noise is injected into the network using adaptive batch normalization, allowing it to adapt to different noise levels and generate more accurate predictions.
  7. 3D U-Net: A 3D U-Net architecture processes the noisy features and generates the final, complete occupancy grid.

By decoupling these two tasks, VisHall3D effectively mitigates feature entanglement and geometric inconsistency.

Datasets

The paper evaluates VisHall3D on two widely-used outdoor Semantic Scene Completion (SSC) datasets:

  • SemanticKITTI [1]: A popular dataset for 3D semantic scene understanding.
  • SSCBench-KITTI-360 [21, 25]: Another benchmark for evaluating SSC methods.

Both datasets use a scene range of 51.2m × 51.2m × 6.4m with a voxel resolution of 0.2m.

Key Insights and Ideas

  • The Power of Decoupling: The core insight is that treating visible and invisible region reconstruction as separate, sequential tasks is more effective than a single-stage approach. This allows specialized modules to handle the distinct challenges of each.
  • Geometric Priors Matter: Using an unsigned distance function to represent the visual frontier, rather than a simple binary mask, provides a richer geometric signal that significantly improves reconstruction accuracy.
  • Intelligent Hallucination: The denoising MAE approach for invisible regions is key. By learning to denoise and reconstruct from noisy inputs, the model can generate more plausible and consistent geometries for unseen parts of the scene.
  • Balancing Noise: The study on noise in OcclusionMAE reveals that the model is sensitive to horizontal noise, suggesting that careful tuning of noise parameters is crucial for optimal performance.

Key Results

VisHall3D achieves state-of-the-art performance on both SemanticKITTI and SSCBench-KITTI-360.

  • SemanticKITTI: Achieves an mIoU of 17.46%, outperforming existing monocular methods.
  • SSCBench-KITTI-360: Achieves an mIoU of 20.95%, surpassing the previous best method CGFormer by 0.90% and even outperforming some early LiDAR-based methods.
  • Qualitative Results: Visual comparisons show VisHall3D generating more accurate and complete scene reconstructions, particularly in occluded areas.
  • Efficiency: Despite its two-stage nature, VisHall3D maintains comparable parameters to other SOTA methods while achieving superior performance and running faster than CGFormer and Monoscene.

Limitations and Future Work

  • Visibility Threshold: The paper notes that while a threshold of 3.5m for visibility relaxation works best, larger thresholds can introduce noise and harm performance. Finding the optimal threshold might be scene-dependent.
  • Noise Sensitivity: The ablation study highlights that the model is more sensitive to horizontal noise than depth noise, suggesting that further research into robust noise handling could be beneficial.
  • Future Directions: The authors suggest that their framework paves the way for more accurate and reliable scene understanding in autonomous driving and other applications, implying potential for further improvements and broader use cases.

Summary updated

BibTeX
@inproceedings{lu2025vishall,
  author = {Haoang Lu and Yuanqi Su and Xiaoning Zhang and Longjun Gao and Yu Xue and Le Wang},
  title = {{VisHall3D: Monocular Semantic Scene Completion from Reconstructing the Visible Regions to Hallucinating the Invisible Regions}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2025},
}
Figure