Abstract
Inferring the 3D geometry and the semantic meaning of surfaces, which are occluded, is a very challenging task. Recently, a first end-to-end learning approach has been proposed that completes a scene from a single depth image. The approach voxelizes the scene and predicts for each voxel if it is occupied and, if it is occupied, the semantic class label. In this work, we propose a two stream approach that leverages depth information and semantic information, which is inferred from the RGB image, for this task. The approach constructs an incomplete 3D semantic tensor, which uses a compact three-channel encoding for the inferred semantic information, and uses a 3D CNN to infer the complete 3D semantic tensor. In our experimental evaluation, we show that the proposed two stream approach substantially outperforms the state-of-the-art for semantic scene completion.
Introduction
A fundamental challenge in robotic perception is understanding the full 3D structure of an environment from a limited viewpoint. When a sensor observes a scene, many surfaces are hidden behind others—a phenomenon known as occlusion. For a robot to navigate safely or interact with objects, it must be able to infer the geometry and semantic identity of these occluded regions.
This task, termed 3D Semantic Scene Completion (SSC), aims to transform a single RGB-D image into a complete 3D voxel grid. Each voxel in this grid is classified as either empty space or a specific object category (e.g., chair, table, wall). Previous state-of-the-art methods relied primarily on geometric data from depth maps. However, geometry alone is often ambiguous; a flat horizontal surface could represent a floor, a bed, or a tabletop. By neglecting the RGB stream, these models discard critical cues like color and texture that are essential for accurate object identification.
Methodology
The proposed two-stream architecture processes geometric and semantic information in parallel to overcome the limitations of depth-only approaches.
2D Semantic Stream and 3D Projection
The system first processes the RGB image through a 2D CNN to generate a pixel-wise semantic segmentation map. This map identifies "what" objects are present. Using the corresponding depth map, these 2D labels are projected into 3D space, creating an "incomplete" 3D semantic tensor where only visible surfaces are labeled.
Compact 3-Channel Encoding
Representing 3D semantic data efficiently is a significant challenge. One-hot encoding is memory-prohibitive, while single-channel integer encoding introduces artificial numerical proximity between unrelated classes. To balance memory efficiency and representational power, the authors employ a 3-channel linear encoding. This maps semantic classes into a 3D vector space, avoiding the pitfalls of integer labels while remaining computationally feasible.
3D CNN Inference
The incomplete semantic volume serves as input to a 14-layer 3D CNN. This network utilizes dilated convolutions to achieve a large receptive field, allowing it to leverage global spatial context—such as the typical relationship between floors and furniture—to "hallucinate" occluded voxels and refine semantic labels across the entire volume.
Experimental Evaluation
The model was evaluated on the NYUv2 dataset, using both raw Kinect depth maps and cleaner CAD-based depth maps.
Key Results
The two-stream approach achieved a new state-of-the-art in semantic scene completion, outperforming previous methods by up to 9.4% on the NYU dataset. Specifically, the model achieved 46.2% IoU on NYU CAD and 34.1% IoU on NYU Kinect for the semantic completion task. Qualitative analysis demonstrates that the model produces significantly cleaner 3D reconstructions with superior object differentiation compared to depth-only baselines.
Conclusion
By integrating 2D semantic priors from RGB images with 3D geometric data, this two-stream approach effectively decouples the tasks of object recognition and spatial completion. While the results are promising, future work may explore deeper 3D architectures and more efficient spatial representations to further improve inference speed and accuracy in real-world robotic applications.
Summary updated