Abstract
Monocular Semantic Occupancy Prediction aims to infer the complete 3D geometry and semantic information of scenes from only 2D images. It has garnered significant attention, particularly due to its potential to enhance the 3D perception of autonomous vehicles. However, existing methods rely on a complex cascaded framework with relatively limited information to restore 3D scenes, including a dependency on supervision solely on the whole network's output, single-frame input, and the utilization of a small backbone. These challenges, in turn, hinder the optimization of the framework and yield inferior prediction results, particularly concerning smaller and long-tailed objects. To address these issues, we propose MonoOcc. In particular, we (i) improve the monocular occupancy prediction framework by proposing an auxiliary semantic loss as supervision to the shallow layers of the framework and an image-conditioned cross-attention module to refine voxel features with visual clues, and (ii) employ a distillation module that transfers temporal information and richer knowledge from a larger image backbone to the monocular semantic occupancy prediction framework with low cost of hardware. With these advantages, our method yields state-of-the-art performance on the camera-based SemanticKITTI Scene Completion benchmark.
Introduction
In autonomous driving, understanding the 3D environment is paramount. While LiDAR provides precise 3D data, its high cost makes monocular solutions—predicting 3D semantic occupancy from a single camera—highly attractive. This task, also known as Semantic Scene Completion (SSC), involves predicting the occupancy and semantic class (e.g., car, sidewalk, vegetation) of 3D voxels.
Despite its potential, monocular SSC faces significant hurdles. Mapping 2D pixels to 3D space is an ill-posed problem where small depth estimation errors lead to large reconstruction inaccuracies. Furthermore, these networks are often deep and cascaded, making optimization difficult when supervised only by the final 3D output. Finally, information scarcity from single-frame inputs and small backbones often leads to poor performance on small or rare "long-tailed" objects.
Methodology
MonoOcc introduces two primary innovations to overcome these challenges: an improved monocular architecture and a privileged distillation module.
Improved Monocular Architecture
To refine the 3D reconstruction, MonoOcc employs an Image-Conditioned Cross-Attention mechanism. Instead of relying solely on noisy initial depth estimates, the model "looks back" at the original 2D image features to refine voxel representations with direct visual clues. Additionally, a 2D Semantic Auxiliary Loss is introduced to provide intermediate supervision to the 2D backbone, ensuring it learns more robust features for the subsequent 3D tasks.
Privileged Distillation Module
The core of MonoOcc's success lies in its teacher-student distillation strategy. A Privileged Branch (Teacher) is trained using a massive pre-trained backbone (InternImage-XL) and multiple temporal frames (video) to capture superior 3D geometry. During training, a lightweight Monocular Branch (Student), using a standard ResNet-50 and a single frame, is forced to mimic the teacher's 3D feature distributions via Kullback-Leibler (KL) divergence. This allows the student to inherit temporal and spatial wisdom without the computational overhead of video processing or massive backbones during inference.
Experiments and Results
MonoOcc was evaluated on the SemanticKITTI benchmark, with the privileged branch pre-trained on a large-scale aggregate of ~200,000 images from datasets like Mapillary Vistas and KITTI-360.
MonoOcc-L achieved a state-of-the-art 15.63 mIoU on the SemanticKITTI test set, significantly outperforming previous leaders like VoxFormer (12.20) and OccFormer (12.32). The model demonstrated exceptional gains in challenging categories: a +131.82% relative improvement on small objects and a +92.74% boost for long-tailed objects compared to the baseline. Even the efficient "Student" version, MonoOcc-S, outperformed previous SOTA models while maintaining a lower memory footprint (8G vs 12G+).
Conclusion
MonoOcc demonstrates that monocular semantic occupancy prediction can be significantly improved by leveraging temporal and large-scale knowledge through distillation. By treating 3D completion as a refinement task guided by 2D visual clues, the model effectively addresses depth uncertainty and information scarcity. While the multi-stage training process is computationally intensive, the resulting inference model is both efficient and highly accurate, particularly for the small and rare objects critical to safe autonomous navigation.
Summary updated