Weakly Supervised Learning of Indoor Geometry by Dual Warping

Pulak Purkait, Ujwal Bonde, Christopher Zach

Introduction

Understanding the 3D geometry of indoor scenes is a fundamental requirement for robotics and augmented reality. A key challenge in this domain is 3D scene completion: the ability to infer the geometry of regions occluded from the camera's current viewpoint. Traditional methods often rely on 3D voxel representations, which are computationally intensive and typically yield low-resolution results. Furthermore, training these models usually necessitates large-scale, manually labeled 3D datasets.

This work introduces a novel approach to high-resolution scene completion using only 2D depth images and weak supervision. By treating completion as a 2D image-inpainting problem in depth space, the proposed Depth-Flow-Net achieves superior resolution and processing speed compared to volumetric methods.

Input depth (a) and the completed scene (b) illustrating the goal of filling in occluded regions from a single depth image.

Methodology

The primary contribution of this research is the Dual Warping training strategy, which enables self-supervised learning of scene geometry.

Dual Warping for Synthetic Supervision

To train a neural network for depth completion, pairs of incomplete and ground-truth depth maps are required. The authors generate these pairs through a three-step process:

  1. Reference Depth ($O$): A complete depth map is selected.
  2. Forward Warping ($P$): The depth map is projected to a new, randomized camera pose, creating realistic occlusions where foreground objects mask the background.
  3. Backward Warping ($\tilde{O}$): The occluded view is warped back to the original pose.

This process yields an image ($\tilde{O}$) with realistic holes that is perfectly aligned with the original complete image ($O$), providing a dense supervision signal without manual labeling.

Overview of the depth prediction strategy (a) and an example of the generated training pairs (b).

Depth-Flow-Net Architecture

The Depth-Flow-Net utilizes a U-Net architecture to predict 2D displacement fields (flow) rather than direct depth values. By predicting where information can be sampled from the existing depth map to fill a hole, the network effectively "copy-pastes" geometric features. This approach preserves the sharp edges and structural integrity of the scene, avoiding the blurring artifacts common in regression-based depth prediction.

Experimental Results

The method was evaluated on the SUNCG synthetic dataset and the NYU Depth v2 real-world dataset. The dual warping strategy allowed for a 25-fold augmentation of the training data.

Performance Metrics

Depth-Flow-Net demonstrates significant improvements over baseline methods:

  • Accuracy: On the NYU dataset, the model achieved a median error of 0.06m, compared to 0.20m for the best-performing semantic-based alternative.
  • Inference Speed: The model is highly efficient, processing a single image in 0.7 seconds on a CPU. This is orders of magnitude faster than optimization-based methods like Mumford-Shah inpainting, which can take over 100 seconds.
  • Qualitative Fidelity: The network successfully recovers fine geometric details, such as thin chair legs and handles, which are often lost in voxel-based completions.

Discussion and Limitations

While Depth-Flow-Net provides a robust solution for indoor geometry completion, certain challenges remain. "Depth spilling" can occur when background values are incorrectly assigned to foreground silhouettes. Additionally, the flow-based mechanism is limited when the required depth information is not present anywhere in the input image. Future research will explore the integration of volumetric fusion to combine multiple completed views into a globally consistent 3D model.

Summary updated

BibTeX
@inproceedings{purkait2018weakly,
  author = {Pulak Purkait and Ujwal Bonde and Christopher Zach},
  title = {{Weakly Supervised Learning of Indoor Geometry by Dual Warping}},
  booktitle = {3DV},
  year = {2018},
  doi = {10.1109/3dv.2018.00089},
}
Figure