MonoOcc: Digging into Monocular Semantic Occupancy Prediction

Yupeng Zheng, Xiang Li, Pengfei Li, Yuhang Zheng, Bu Jin, Chengliang Zhong, Xiaoxiao Long, Hao Zhao, Qichao Zhang

TL;DR

MonoOcc is a novel framework that significantly boosts the accuracy of 3D semantic occupancy prediction from a single 2D image. By using a "privileged" teacher model—which has access to multiple video frames and a massive pre-trained backbone—to train a lightweight student model via distillation, MonoOcc achieves state-of-the-art results on the SemanticKITTI benchmark, particularly for small and rare objects.

What Problem Does This Paper Address?

In the world of autonomous driving, understanding the 3D environment is crucial. While LiDAR provides precise 3D data, it is expensive. Monocular Semantic Occupancy Prediction (also known as Semantic Scene Completion or SSC) aims to do the same using just a single camera. The goal is to predict which 3D "voxels" (3D pixels) in a scene are occupied and what they represent (e.g., a car, a tree, or a sidewalk).

However, current monocular methods face three major hurdles:

  1. Depth Uncertainty: Mapping 2D pixels to 3D space is mathematically "ill-posed." Small errors in depth estimation lead to massive errors in the 3D reconstruction.
  2. Optimization Difficulties: These networks are "long and cascaded" (2D backbone $\rightarrow$ 3D transformer $\rightarrow$ 3D head). Training them solely on the final 3D output makes it hard for the early 2D layers to learn effectively.
  3. Information Scarcity: Most models use a single frame and a small image backbone, causing them to miss small objects (like traffic signs) or rare "long-tailed" classes (like trucks).

Methods

The authors introduce MonoOcc, which tackles these issues through two main innovations:

1. Improved Architecture

  • Image-Conditioned Cross-Attention: Instead of just relying on a noisy depth map to place features in 3D, the model uses a cross-attention mechanism to "look back" at the original 2D image. This allows the model to use visual clues to refine and correct the 3D voxel features.
  • 2D Semantic Auxiliary Loss: To help the 2D backbone learn better features, the authors add a "shortcut" supervision. They force the early layers to predict 2D semantic maps, providing a clearer gradient for optimization.

2. The Privileged Distillation Module

This is the "secret sauce" of the paper. The authors create two branches:

  • The Privileged Branch (Teacher): This version is "overpowered." It uses a massive backbone (InternImage-XL with 350M parameters) and looks at multiple temporal frames (video) to get a better sense of the 3D geometry.
  • The Monocular Branch (Student): This is the lightweight version intended for real-time use, using a standard ResNet-50 and a single frame.

During training, the student is forced to mimic the 3D feature distributions of the teacher using Kullback-Leibler (KL) Divergence. This allows the student to "inherit" the spatial and temporal wisdom of the larger model without needing the extra frames or the massive backbone during actual deployment.

Datasets

  • SemanticKITTI: The primary benchmark for evaluation, consisting of outdoor driving scenes with 19 semantic classes.
  • Large-scale Pre-training: To train the "Privileged" backbone, the authors used a massive aggregate of ~200,000 images from Mapillary Vistas, KITTI-360, BDD100K, Cityscapes, and nuImages.

Key Insights and Ideas

  • Temporal Wisdom without Temporal Cost: One of the most clever insights is that you don't need video at inference time to benefit from temporal information. By distilling a multi-frame teacher into a single-frame student, the student learns to "hallucinate" the occluded areas that the teacher saw in previous frames.
  • Refinement over Generation: Previous models treated 3D completion as a generative task (filling in the blanks). MonoOcc treats it as a refinement task, using the 2D image as a persistent ground truth to correct geometric drift.
  • Solving the "Long Tail": By pre-training the teacher on diverse datasets (like Mapillary Vistas, which has 124 classes), the model becomes much better at recognizing rare objects like trucks or motorcycles that are sparse in the standard KITTI dataset.

Key Results

MonoOcc achieved State-of-the-Art (SOTA) performance on the SemanticKITTI test set:

  • Overall Performance: MonoOcc-L reached 15.63 mIoU, significantly outperforming previous leaders like VoxFormer (12.20) and OccFormer (12.32).
  • Small Objects: The model showed massive relative gains: +131% on bicycles and +92% on traffic signs compared to the baseline.
  • Long-tailed Objects: It achieved a +58% boost on trucks and +92% on "other-vehicles."
  • Efficiency: The "Student" version (MonoOcc-S) achieves better results than previous SOTAs while maintaining a low memory footprint (8G vs 12G+ for larger models) during inference.

Limitations and Future Work

  • Training Complexity: While the inference is efficient, the training process is heavy, requiring a multi-stage pipeline (pre-training the large backbone, then training the teacher, then distilling to the student).
  • Depth Dependency: Although the image-conditioned attention helps, the model still starts with a depth-based query. If the initial depth estimation is catastrophically wrong, the refinement has limits.
  • Future Directions: The authors suggest that exploring even more diverse data sources for the privileged branch could further improve the "long-tail" recognition of rare 3D objects.

Summary updated

Abstract

Monocular Semantic Occupancy Prediction aims to infer the complete 3D geometry and semantic information of scenes from only 2D images. It has garnered significant attention, particularly due to its potential to enhance the 3D perception of autonomous vehicles. However, existing methods rely on a complex cascaded framework with relatively limited information to restore 3D scenes, including a dependency on supervision solely on the whole network's output, single-frame input, and the utilization of a small backbone. These challenges, in turn, hinder the optimization of the framework and yield inferior prediction results, particularly concerning smaller and long-tailed objects. To address these issues, we propose MonoOcc. In particular, we (i) improve the monocular occupancy prediction framework by proposing an auxiliary semantic loss as supervision to the shallow layers of the framework and an image-conditioned cross-attention module to refine voxel features with visual clues, and (ii) employ a distillation module that transfers temporal information and richer knowledge from a larger image backbone to the monocular semantic occupancy prediction framework with low cost of hardware. With these advantages, our method yields state-of-the-art performance on the camera-based SemanticKITTI Scene Completion benchmark. Codes and models can be accessed at https://github.com/ucaszyp/MonoOcc
BibTeX
@inproceedings{zheng2024monoocc,
  author = {Yupeng Zheng and Xiang Li and Pengfei Li and Yuhang Zheng and Bu Jin and Chengliang Zhong and Xiaoxiao Long and Hao Zhao and Qichao Zhang},
  title = {{MonoOcc: Digging into Monocular Semantic Occupancy Prediction}},
  booktitle = {IEEE International Conference on Robotics and Automation},
  year = {2024},
  doi = {10.1109/icra57147.2024.10611261},
}

Loading related content...

Figure