TL;DR
This paper introduces RDFC-GAN, a novel two-branch network for indoor depth completion that effectively fuses RGB and depth data. By leveraging a Manhattan-world assumption for geometric priors and a CycleGAN for texture synthesis, RDFC-GAN significantly improves depth map accuracy and detail, especially in challenging indoor environments with large missing regions.
What Problem Does This Paper Address?
Indoor depth sensors, like those found in AR/VR or robotics, often produce incomplete depth maps. This is due to limitations with transparent or reflective surfaces, sensor range, and oblique angles. These "holes" in the depth data are problematic for downstream tasks like 3D reconstruction or navigation. While existing methods can fill in sparse depth data, they struggle with the large, contiguous missing regions common in indoor scenes. Furthermore, typical training strategies for depth completion often use random sampling of sparse points, which doesn't accurately reflect the real-world missing patterns in indoor RGB-D data, leading to potentially misleading evaluations.
Methods
RDFC-GAN is a two-branch, end-to-end network designed to tackle these challenges:
- Manhattan-Constraint Network (MCN) Branch: This branch leverages the "Manhattan world assumption," which states that most indoor surfaces are aligned with three dominant orthogonal directions (floor, ceiling, walls). It uses a pre-trained segmentation network to identify these regions and a U-Net to generate a normal map that adheres to this assumption. This geometric prior helps in predicting depth, especially in planar regions. The branch then uses a convolutional encoder-decoder to output a local depth map and a confidence map.
- RGB-Depth Fusion CycleGAN (RDFC-GAN) Branch: This branch uses a generative adversarial network (GAN) approach inspired by CycleGAN. It takes the RGB image as a condition and the depth latent vector from the MCN branch to generate a fused, dense depth map and a confidence map. The CycleGAN component ensures that the generated depth maps retain texture and detail by translating between RGB and depth domains while maintaining consistency.
- Weighted Adaptive Instance Normalization (W-AdaIN): This module is crucial for fusing information between the two branches. It adaptively injects depth features into the RGB branch's feature maps at intermediate stages, allowing the depth information to guide the texture synthesis process while preserving semantic features from the RGB image.
- Confidence Fusion Head: This final module combines the outputs from the MCN and RDFC-GAN branches, weighted by their respective confidence maps, to produce the final, robust depth prediction.
- Pseudo Depth Map Training: To better mimic real-world indoor depth missing patterns, the authors introduce five synthetic masking strategies (highlight masking, black masking, graph-based segmentation, semantic masking, and semantic XOR masking) to generate more realistic "pseudo" depth maps for training.
Datasets
The authors evaluate their method on two widely-used indoor RGB-D datasets:
- NYU-Depth V2: This dataset contains RGB images and corresponding depth maps from a Kinect sensor. The authors train on a combination of labeled and unlabeled data and test on the labeled test set.
- SUN RGB-D: This dataset offers a more diverse collection of RGB-D images from multiple sensors, making it suitable for evaluating model generalization. The authors use the refined depth maps derived from multiple frames as ground truth.
Key Insights and Ideas
- Manhattan World Assumption for Indoor Depth: Explicitly incorporating the Manhattan world assumption into depth completion is a novel approach that significantly benefits indoor scenes, where this geometric regularity is prevalent.
- CycleGAN for Texture-Rich Depth: Using a CycleGAN for RGB-depth fusion is effective in generating detailed and textured depth maps, overcoming the blurriness often seen in purely regression-based methods.
- Pseudo Depth Maps Matter: Randomly sampling sparse points is insufficient for training indoor depth completion models. The proposed pseudo depth map generation methods better simulate real-world missing patterns, leading to more robust models.
- W-AdaIN for Effective Fusion: The W-AdaIN module is key to successfully integrating geometric priors from the MCN branch with the texture synthesis capabilities of the RDFC-GAN branch.
Key Results
- State-of-the-Art Performance: RDFC-GAN achieves state-of-the-art results on both NYU-Depth V2 and SUN RGB-D datasets across various metrics, including RMSE, Rel, and $\delta_{th}$.
- NYU-Depth V2: In the most realistic "R ⇒ T" setting (raw depth to completed depth), RDFC-GAN achieves an RMSE of 0.120 and a Rel of 0.012, outperforming previous methods.
- SUN RGB-D: RDFC-GAN also demonstrates superior performance on the more challenging SUN RGB-D dataset, with an RMSE of 0.214.
- Qualitative Improvements: Visualizations show that RDFC-GAN generates more accurate and textured depth maps, particularly in regions with subtle objects and large missing areas, compared to baseline methods.
- Point Cloud Evaluation: Point cloud metrics (Chamfer Distance and F1 score) further validate RDFC-GAN's superior geometric accuracy and robustness.
- Object Detection Improvement: Using RDFC-GAN's completed depth maps as input for 3D object detection tasks on SUN RGB-D led to moderate improvements in mAP for standard detectors like VoteNet and H3DNet.
Limitations and Future Work
- Manhattan World Assumption: While beneficial for indoor scenes, the Manhattan world assumption might not be as effective in highly unstructured or outdoor environments.
- Computational Cost: The two-branch structure and CycleGAN component likely increase computational complexity compared to simpler methods.
- Future Work: The authors suggest that their method could be extended to handle more complex indoor scenes or potentially adapted for outdoor depth completion by modifying the Manhattan-constraint module. The paper also hints at the potential for using the learned confidence maps for downstream tasks.
Summary updated