Introduction
Single Image Depth Estimation (SIDE) has seen rapid progress with the advent of transformer-based architectures. However, these models frequently struggle with high-frequency details and complex occlusions. Even state-of-the-art models like MiDaS and DPT often produce depth maps characterized by three primary flaws: blurry boundaries that create "halo" artifacts, the disappearance of thin structures like wires or fingers, and the "hole problem," where background regions visible through foreground gaps (e.g., the space between a person's arm and torso) are incorrectly assigned foreground depth values.
This paper addresses these limitations by leveraging a widely available resource: high-quality image masks. With the proliferation of commercial background removal tools and advanced segmentation models, obtaining a precise mask is often easier than obtaining a precise depth map. The authors propose a framework that uses these masks as a geometric guide to "clean up" and refine existing depth predictions.
Proposed Method: Layered Refinement
The central innovation of this work is the move away from simple input concatenation. While one might expect that feeding a mask as an additional channel to a neural network would solve boundary issues, the authors demonstrate that networks often fail to respect the hard edges of the mask, still resulting in blurred transitions.
Two-Stage Composition
To enforce geometric sharpness, the framework adopts a layered approach:
- Layered Completion: The system utilizes a modified Dense Prediction Transformer (DPT) trained for depth inpainting and outpainting. Given an initial "noisy" depth map and a guidance mask ($M$), the model separately predicts complete depth layers for the foreground (the region within $M$) and the background (the region within $1-M$). This process treats the foreground and background as overlapping sheets, filling in missing information behind and in front of the respective layers.
- Sharp Composition: The final depth map is generated by merging these two predicted layers using the original high-quality mask. Because the mask itself dictates the boundary of the merge, the resulting depth map inherits the mask's exact contours, ensuring razor-sharp edges and eliminating "flying pixels."
Self-Supervised Training via Perturbation
A significant challenge in developing this framework was the scarcity of datasets containing synchronized RGB images, high-fidelity depth maps, and precise instance masks. To overcome this, the authors designed a self-supervised data generation scheme. They utilized existing RGB-D data and applied controlled "perturbations" to the ground-truth depth to simulate common SIDE errors:
- Dilation and Erosion: To mimic the loss of thin structures.
- Gaussian Blurring: To simulate the typical lack of edge sharpness in standard models.
- Human Hole Perturbation: A specialized script that identifies gaps in human silhouettes and fills them with incorrect depth values, forcing the model to learn how to identify and "re-open" these background holes during the refinement process.
Experimental Evaluation
The model was evaluated on a mix of synthetic and real-world data, including Hypersim (photorealistic indoor scenes), TartanAir (complex outdoor environments), and "in-the-wild" images sourced from Unsplash and Pixabay. For real-world testing, masks were generated using off-the-shelf tools like remove.bg.
Quantitative Results
The authors introduced a new metric, the Relative Refinement Ratio ($R^3$), to evaluate the effectiveness of the refinement. $R^3$ measures the ratio of pixels that were improved by the refinement process compared to those that were degraded. On the TartanAir dataset, the method achieved an $R^3$ of 16.569, indicating that for every pixel the model negatively impacted, it improved more than 16 others.
Qualitative Impact
Visually, the refinement produces significant improvements in downstream applications. In 3D point cloud reconstructions, the elimination of blurry edges removes the "flying pixel" artifacts that typically bridge foreground and background objects. In computational photography, such as Bokeh effects, the transition between the sharp subject and the blurred background becomes professional-grade, avoiding the color bleeding common in standard depth-based blurring.
Discussion and Limitations
While the layered refinement strategy is highly effective, it is inherently dependent on the quality of the guidance mask. If the input mask is inaccurate—for instance, if it fails to capture a part of an object—the refinement process will not correct that specific geometric error. Furthermore, the method is designed to fix boundaries and structural holes; it does not necessarily correct absolute depth estimation errors within the interior of a mask if the underlying SIDE model's prediction is fundamentally skewed.
Conclusion
By reframing depth refinement as a layered completion task guided by masks, this paper provides a robust solution to the persistent issues of blurriness and occlusion errors in SIDE. The approach demonstrates that leveraging the strengths of segmentation models can significantly compensate for the current weaknesses in depth estimation, moving us closer to high-fidelity 3D scene understanding from single images.
Summary updated