Layered Depth Refinement with Mask Guidance

Soo Ye Kim, Jianming Zhang, Simon Niklaus, Yifei Fan, Simon Chen, Zhe Lin, Munchurl Kim

This paper, presented at CVPR 2022, introduces a novel approach to one of the most persistent problems in computer vision: getting sharp, accurate depth maps from a single image. While Single Image Depth Estimation (SIDE) has come a long way, it still struggles with "messy" details—blurry boundaries, missing thin structures like wires, and incorrect depth values in the gaps between objects (like the space between a person's arm and their body).

TL;DR

The authors propose a mask-guided, layered refinement framework that uses high-quality object masks to fix the errors of single-image depth models. By decomposing the depth map into separate layers and training on a clever self-supervised synthetic dataset, they achieve state-of-the-art boundary sharpness and "hole" correction for applications like Bokeh effects and 3D photography.

What Problem Does This Paper Address?

Single Image Depth Estimation (SIDE) models are powerful but imperfect. They often produce "halo" effects around objects, fail to resolve thin structures, and struggle with "isolated holes"—background regions visible through foreground gaps.

The authors observe a key opportunity: while high-quality ground-truth depth is hard to get, high-quality masks are now ubiquitous. Tools like Photoshop, remove.bg, and modern segmentation models can easily isolate objects. This paper asks: Can we use these easily obtained masks to "guide" and fix the messy depth maps produced by SIDE models?

Methods

The core innovation is a layered refinement strategy. Instead of simply feeding a mask into a neural network as an extra input channel (which the authors show still results in blurry edges), they treat the foreground and background as distinct layers.

  1. Layered Decomposition: The initial depth map is split into two layers based on the mask ($M$) and the inverse mask ($1-M$).
  2. Inpainting and Outpainting: The model refines each layer separately. For the foreground layer, it "outpaints" depth values slightly beyond the mask boundary. For the background layer, it "inpaints" values to fill in the regions covered by the foreground.
  3. Composition: The two refined layers are merged using the original high-quality mask. This ensures the final depth map has boundaries that are pixel-perfectly aligned with the mask.
Overview of the two-stage training strategy. Stage I focuses on completion, while Stage II focuses on refinement and perturbation removal.

To train this without a massive dataset of perfect depth/mask pairs, the authors developed a self-supervised data generation scheme. They take two random RGB-D patches, composite them using an arbitrary mask, and then apply "perturbations" to the depth—simulating the blur, erosion, and "hole" errors typical of SIDE models. The network then learns to recover the original, clean depth map from this corrupted version.

Datasets

  • Training: The model was trained on diverse indoor and outdoor natural RGB-D images.
  • Evaluation: The authors used Hypersim and TartanAir, two synthetic datasets known for having dense, highly accurate ground-truth depth and instance segmentation maps, which are essential for measuring fine boundary errors.
  • Qualitative Testing: Real-world images from Unsplash and Pixabay were used to demonstrate the model's performance "in the wild."

Key Insights and Ideas

  • Layering is Mandatory: Simply concatenating a mask to the input isn't enough. The network needs the structural constraint of layering to truly respect the mask boundaries.
  • The "Human Hole" Perturbation: The authors specifically designed a perturbation to simulate the way SIDE models fail in the gaps between human limbs. By training the model on these synthetic "holes," it learns how to correctly identify and fill background depth in those tricky regions.
  • Architecture Matters: The model uses a Dense Prediction Transformer (DPT) backbone but adds a low-level encoder to ensure that the fine details of the initial depth map aren't lost during the refinement process.
Figure 1. Our layered depth refinement result on an initial predic- tion by DPT [31]. Aided by a high-quality mask M generated with an auto-masking tool [33], our method is able to accurately refine mask boundaries and correct depth values in isolated hole regions between body parts. Regions in M and 1 −M are refined and inpainted/outpainted separately with our layered approach.

Key Results

The method significantly outperforms existing depth refinement baselines across multiple metrics:

  • Boundary Accuracy: On the TartanAir dataset, the proposed method achieved a Relative Refinement Ratio ($R^3$) of 16.569 when refining MiDaS v2.1 outputs, indicating a massive improvement in pixel-wise accuracy compared to the base model.
  • Mask Boundary Error (MBE): The model consistently reduced MBE, proving it effectively "snaps" depth values to the correct side of the object boundary.
  • Downstream Impact: The authors demonstrate that their refined depth maps lead to much cleaner Bokeh effects (no more blurry halos around hair or arms) and more accurate 3D point cloud reconstructions with fewer "flying pixels."

Limitations and Future Work

  • Mask Dependency: The refinement quality is strictly bounded by the quality of the input mask. If the mask is poor, the refinement will be too.
  • Internal Errors: While the model is excellent at fixing boundaries and holes, it is less likely to correct fundamental depth errors deep inside a large, homogeneous object if the initial SIDE model was significantly off.
  • Future Directions: The authors suggest this layered refinement approach could be extended to other dense prediction tasks, such as refining surface normals or optical flow.

For more details, code, and examples, visit the project page.

Summary updated

Abstract

Depth maps are used in a wide range of applications from 3D rendering to 2D image effects such as Bokeh. However, those predicted by single image depth estimation (SIDE) models often fail to capture isolated holes in objects and/or have inaccurate boundary regions. Meanwhile, high-quality masks are much easier to obtain, using commercial auto-masking tools or off-the-shelf methods of segmentation and matting or even by manual editing. Hence, in this paper, we formulate a novel problem of mask-guided depth refinement that utilizes a generic mask to refine the depth prediction of SIDE models. Our framework performs layered refinement and inpainting/outpainting, decomposing the depth map into two separate layers signified by the mask and the inverse mask. As datasets with both depth and mask annotations are scarce, we propose a self-supervised learning scheme that uses arbitrary masks and RGB-D datasets. We empirically show that our method is robust to different types of masks and initial depth predictions, accurately refining depth values in inner and outer mask boundary regions. We further analyze our model with an ablation study and demonstrate results on real applications. More information can be found at https://sooyekim.github.io/MaskDepth/ .
BibTeX
@inproceedings{kim2022layered,
  author = {Soo Ye Kim and Jianming Zhang and Simon Niklaus and Yifei Fan and Simon Chen and Zhe Lin and Munchurl Kim},
  title = {{Layered Depth Refinement with Mask Guidance}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2022},
}
Figure