TL;DR
This paper introduces SPAWN, a novel 3D Convolutional Neural Network (CNN) for Semantic Scene Completion (SSC) that leverages 2D semantic segmentation priors from RGB and surface normal inputs to guide depth-based structural information. To combat the scarcity of large-scale 3D datasets, they propose a novel 3D data augmentation strategy that significantly improves performance and achieves state-of-the-art results on benchmark datasets.
What Problem Does This Paper Address?
Semantic Scene Completion (SSC) is a crucial task in computer vision that aims to reconstruct the 3D geometry of a scene and simultaneously label each voxel with its semantic category, including occluded regions. This is vital for applications like robotics and augmented reality. A major hurdle in this field is the lack of sufficiently large, fully labeled real-world 3D datasets, which are essential for training data-hungry deep learning models. While 2D computer vision has benefited greatly from data augmentation techniques, directly applying these to 3D SSC is challenging due to the inherent differences in dimensionality between input (2D) and output (3D) data, and the complex propagation of changes in 3D representations like Truncated Signed Distance Functions (TSDF).
Methods
The proposed solution, SPAWN, is a multimodal 3D CNN that fuses information from multiple sources.
- 2D Semantic Priors: A lightweight, bimodal 2D segmentation network takes RGB images and surface normals as input. This network is a simplified version of RDFNet, using pre-trained ResNet-101 backbones. The output probabilities from this network are projected into a low-resolution 3D voxel volume, providing semantic guidance.
- 3D Structural Information: The depth map from the RGB-D input is projected into a high-resolution 3D volume and encoded using the F-TSDF representation.
- SPAWN Architecture: The core of SPAWN is a novel 3D CNN that fuses the semantic priors and structural information. It utilizes a new memory-saving building block called BN-DDR (Batch-Normalized Dimensional Decomposition Residual), which is a more memory-efficient alternative to standard DDR blocks, allowing for larger mini-batches and faster training.
- 3D Data Augmentation: The paper introduces a novel 3D data augmentation strategy specifically designed for SSC. Instead of augmenting the 2D inputs, the augmentation is applied directly to the 3D volumes (both input and ground truth) using three simple and fast transformations: random X-axis flipping, random Z-axis flipping, and random X-Z axes swapping. This strategy effectively increases the training data by a factor of eight and can be applied to both training and test times.
- Fusion Strategy: An early/late fusion strategy is employed to combine the semantic and structural information. The 2D semantic priors and the 3D structural data are first processed in separate branches to match their resolutions and channel counts. They are then fused in an encoder-decoder architecture with skip connections, inspired by U-Net.
- Loss Function: A class-weighted cross-entropy loss is used to address class imbalance, with higher weights assigned to less frequent classes.
Datasets
The authors evaluated their approach on three benchmark datasets:
- NYUDv2: A real-world dataset containing RGB-D images.
- NYUCAD: A synthesized dataset derived from NYUDv2, where depth maps are generated from 3D object meshes, aiming to reduce misalignment issues.
- SUNCG: A large-scale synthetic dataset comprising approximately 45,000 3D house models, from which over 130,000 3D snapshots were extracted. RGB images and surface normals were rendered from these models.
Key Insights and Ideas
- 2D Priors for 3D Guidance: Using 2D semantic segmentation priors as guidance for 3D structural data is a highly effective way to improve SSC performance, especially in regions where depth information alone might be ambiguous.
- 3D Data Augmentation is Crucial: The scarcity of 3D data is a significant bottleneck. Applying data augmentation directly in 3D voxel space, rather than on 2D inputs, is a novel and effective solution that significantly boosts performance and generalization.
- Memory Efficiency Matters: The proposed BN-DDR block is a key enabler, allowing for efficient training of complex 3D models on standard hardware (a single 10GB GPU) with reasonable mini-batch sizes.
- Fusion is Key: The early/late fusion strategy effectively combines the complementary information from semantic and structural modalities, leading to more accurate and complete scene reconstructions.
- Oracle Test Reveals Room for Improvement: The oracle test, where perfect 2D priors are used, indicates that further improvements in the quality of 2D semantic segmentation could lead to even better SSC results, suggesting future research directions.
Key Results
- SPAWN, with its proposed data augmentation, consistently surpasses previous state-of-the-art methods on both real and synthetic datasets, often by significant margins.
- On the NYUDv2 dataset, SPAWN achieved an average IoU of 48.0% without pretraining, and when fine-tuned from SUNCG, it achieved 49.9% avg. IoU, outperforming previous methods.
- On the SUNCG dataset, SPAWN achieved 41.3% avg. IoU, surpassing the previous best straightforward solution (CCPNet) by 6.3% (4.7 p.p.) and achieving comparable results to iterative models.
- On the NYUCAD dataset, SPAWN achieved 62.2% avg. IoU, a significant improvement over previous methods.
- The proposed BN-DDR module and data augmentation strategy were shown to provide substantial performance gains in ablation studies.
Limitations and Future Work
- 2D Prior Quality: The oracle test demonstrated that the performance of SPAWN is still limited by the quality of the 2D semantic priors. Future work could focus on improving the 2D segmentation network or exploring better sources of semantic information.
- Iterative Solutions: While SPAWN achieves comparable results to more expensive iterative solutions, the paper acknowledges that iterative approaches might still offer marginal improvements.
- Mode and Resolution Agnosticism: The authors suggest that their data augmentation strategy is mode and resolution agnostic and could be applied to other multimodal or multi-resolution SSC setups.
The paper concludes by introducing SPAWN as a novel and effective solution for 3D semantic scene completion, highlighting the power of fusing semantic priors with structural information and the impact of a well-designed 3D data augmentation strategy.
Summary updated