This paper, presented at CVPR 2022, introduces a novel approach to one of the most persistent problems in computer vision: getting sharp, accurate depth maps from a single image. While Single Image Depth Estimation (SIDE) has come a long way, it still struggles with "messy" details—blurry boundaries, missing thin structures like wires, and incorrect depth values in the gaps between objects (like the space between a person's arm and their body).
TL;DR
The authors propose a mask-guided, layered refinement framework that uses high-quality object masks to fix the errors of single-image depth models. By decomposing the depth map into separate layers and training on a clever self-supervised synthetic dataset, they achieve state-of-the-art boundary sharpness and "hole" correction for applications like Bokeh effects and 3D photography.
What Problem Does This Paper Address?
Single Image Depth Estimation (SIDE) models are powerful but imperfect. They often produce "halo" effects around objects, fail to resolve thin structures, and struggle with "isolated holes"—background regions visible through foreground gaps.
The authors observe a key opportunity: while high-quality ground-truth depth is hard to get, high-quality masks are now ubiquitous. Tools like Photoshop, remove.bg, and modern segmentation models can easily isolate objects. This paper asks: Can we use these easily obtained masks to "guide" and fix the messy depth maps produced by SIDE models?
Methods
The core innovation is a layered refinement strategy. Instead of simply feeding a mask into a neural network as an extra input channel (which the authors show still results in blurry edges), they treat the foreground and background as distinct layers.
- Layered Decomposition: The initial depth map is split into two layers based on the mask ($M$) and the inverse mask ($1-M$).
- Inpainting and Outpainting: The model refines each layer separately. For the foreground layer, it "outpaints" depth values slightly beyond the mask boundary. For the background layer, it "inpaints" values to fill in the regions covered by the foreground.
- Composition: The two refined layers are merged using the original high-quality mask. This ensures the final depth map has boundaries that are pixel-perfectly aligned with the mask.
To train this without a massive dataset of perfect depth/mask pairs, the authors developed a self-supervised data generation scheme. They take two random RGB-D patches, composite them using an arbitrary mask, and then apply "perturbations" to the depth—simulating the blur, erosion, and "hole" errors typical of SIDE models. The network then learns to recover the original, clean depth map from this corrupted version.
Datasets
- Training: The model was trained on diverse indoor and outdoor natural RGB-D images.
- Evaluation: The authors used Hypersim and TartanAir, two synthetic datasets known for having dense, highly accurate ground-truth depth and instance segmentation maps, which are essential for measuring fine boundary errors.
- Qualitative Testing: Real-world images from Unsplash and Pixabay were used to demonstrate the model's performance "in the wild."
Key Insights and Ideas
- Layering is Mandatory: Simply concatenating a mask to the input isn't enough. The network needs the structural constraint of layering to truly respect the mask boundaries.
- The "Human Hole" Perturbation: The authors specifically designed a perturbation to simulate the way SIDE models fail in the gaps between human limbs. By training the model on these synthetic "holes," it learns how to correctly identify and fill background depth in those tricky regions.
- Architecture Matters: The model uses a Dense Prediction Transformer (DPT) backbone but adds a low-level encoder to ensure that the fine details of the initial depth map aren't lost during the refinement process.
Key Results
The method significantly outperforms existing depth refinement baselines across multiple metrics:
- Boundary Accuracy: On the TartanAir dataset, the proposed method achieved a Relative Refinement Ratio ($R^3$) of 16.569 when refining MiDaS v2.1 outputs, indicating a massive improvement in pixel-wise accuracy compared to the base model.
- Mask Boundary Error (MBE): The model consistently reduced MBE, proving it effectively "snaps" depth values to the correct side of the object boundary.
- Downstream Impact: The authors demonstrate that their refined depth maps lead to much cleaner Bokeh effects (no more blurry halos around hair or arms) and more accurate 3D point cloud reconstructions with fewer "flying pixels."
Limitations and Future Work
- Mask Dependency: The refinement quality is strictly bounded by the quality of the input mask. If the mask is poor, the refinement will be too.
- Internal Errors: While the model is excellent at fixing boundaries and holes, it is less likely to correct fundamental depth errors deep inside a large, homogeneous object if the initial SIDE model was significantly off.
- Future Directions: The authors suggest this layered refinement approach could be extended to other dense prediction tasks, such as refining surface normals or optical flow.
For more details, code, and examples, visit the project page.
Summary updated