TL;DR
This paper introduces a novel two-stage framework for robust single image depth estimation (SIDE) that leverages scene understanding to categorize images into "low depth-range" or "high depth-range" categories. This allows for specialized SIDE networks to be applied, significantly improving accuracy and generalization across diverse indoor and outdoor scenes, outperforming previous state-of-the-art methods.
What Problem Does This Paper Address?
Estimating the depth of a scene from a single 2D image (SIDE) is a fundamental but challenging problem in computer vision. A single image can correspond to infinitely many 3D scenes, making it an "ill-posed" problem. Traditional methods rely on geometric cues like perspective and texture. While deep learning has significantly advanced SIDE, current models often struggle with generalization. They are typically trained on specific datasets (e.g., indoor or outdoor) and perform poorly when applied to data from a different domain. This paper tackles two key issues: (1) developing a "blind" SIDE framework that can handle both indoor and outdoor images without prior domain knowledge, and (2) designing an efficient, fully-convolutional network that achieves high accuracy.
Methods
The core contribution is a two-stage robust SIDE framework.
- Stage 1: Scene Understanding: This stage categorizes the input RGB image into one of two depth ranges: "low depth-range" (objects are generally close) or "high depth-range" (objects can be far away). The authors propose two methods for this:
- Scene Classification: Using a pre-trained scene classification network (WideResNet-18 on Places-365) to predict scene categories and then voting to determine the depth range.
- Coarse Depth Estimation: Training a separate, coarse SIDE network on a mix of indoor and outdoor data. The maximum estimated depth from this network is then used to infer the depth range.
- Stage 2: Accurate SIDE: Based on the determined depth range, a specialized SIDE network is applied. The authors introduce DS-SIDENet, a multi-task, encoder-decoder architecture that utilizes depthwise separable convolutions. This design choice significantly improves computational efficiency. DS-SIDENet is trained with two branches:
- Depth Classification Branch: Predicts pixel-wise classification labels for quantized depth.
- Depth Regression Branch: Directly estimates a continuous depth map. The network is optimized using a combined loss function for both branches, but only the classification output is used during inference, which the authors found to be sharper and more accurate. They also propose an improved upsampling layer, "upproj-con," with skip connections.
Datasets
- NYU-Depth-v2: A widely used indoor dataset for training and evaluating the DS-SIDENet.
- ScanNet: A large-scale RGB-D dataset for indoor scene reconstruction, used for training and evaluation in the context of the ROB challenge.
- KITTI: A real-world dataset with street views, used for training and evaluation in the context of the ROB challenge.
- Places-365: Used for pre-training the scene classification network in Stage 1.
The paper focuses on the Robust Vision Challenge (ROB) 2018 benchmark, which involves evaluating SIDE performance on both ScanNet and KITTI datasets, simulating a "blind" scenario.
Key Insights and Ideas
- Scene Understanding is Crucial for Robustness: The paper strongly advocates that explicitly understanding the scene's general depth characteristics (low vs. high range) is key to achieving robust SIDE across diverse domains.
- Specialization Beats Generalization (with a twist): Instead of training one massive network for everything, breaking down the problem into domain-specific (depth-range specific) networks, guided by scene understanding, yields better results.
- Multi-task Learning Improves Feature Representation: Training DS-SIDENet with both depth classification and regression tasks simultaneously leads to better feature extraction in the encoder, even if only one output is used for final prediction.
- Depthwise Separable Convolutions for Efficiency: This architectural choice is critical for making the network fast enough for practical applications without sacrificing accuracy.
- Classification Output is Sharper: The authors observe that the depth map derived from the classification branch of DS-SIDENet is sharper than that from the regression branch, suggesting its utility for more precise depth estimation.
Key Results
- State-of-the-Art Performance: The proposed robust SIDE framework achieved the top rank among submissions to the Robust Vision Challenge (ROB) 2018 on both ScanNet and KITTI datasets.
- Competitive Accuracy on NYU: DS-SIDENet achieved competitive performance on the NYU-Depth-v2 dataset, outperforming the state-of-the-art DORN [10] in terms of accuracy.
- Significant Efficiency Gains: DS-SIDENet is considerably faster than DORN and MS-CRF, predicting a 640x480 image in 150ms on a Tesla V100 GPU, compared to 600ms or 1 second for others.
- Scene Classification vs. Coarse Depth Estimation: Both scene understanding methods performed equally well in the ROB challenge, achieving 100% classification accuracy. However, the paper notes that scene classification is more practical and less sensitive to threshold errors in real-world scenarios.
Limitations and Future Work
- Threshold Sensitivity for Coarse Depth Estimation: The paper highlights that using coarse depth estimation for scene understanding is sensitive to the chosen depth threshold. Incorrect thresholds can lead to performance degradation, especially when mixing indoor and outdoor data.
- External Training Data for Scene Classification: The scene classification approach requires pre-trained models on large external datasets like Places-365, which might not always be available or feasible.
- Fusion of Branches: While the authors mention preliminary results showing that fusing the outputs of the classification and regression branches can further enhance accuracy, this was not a primary focus of the presented work.
- Future Work: The authors suggest exploring fusion strategies for combining the outputs of the different branches of DS-SIDENet to potentially achieve even higher accuracy. They also imply that further improvements could be made by exploring more advanced decoding modules, such as SAM (Stacked Atrous Multiscale) for high depth-range images.
Summary updated