Super-Resolution for Monocular Depth Estimation With Multi-Scale Sub-Pixel Convolutions and a Smoothness Constraint

Shiyu Zhao, Lin Zhang, Ying Shen, Shengjie Zhao, Huijuan ZhangIEEE Access 2019 · OpenReview

Abstract

Depth estimation from a monocular image is of paramount importance in various vision tasks, such as obstacle detection, robot navigation, and 3D reconstruction. However, how to get an accurate depth map with clear details and a fine resolution remains an unresolved issue. As an attempt to solve this problem, we exploit image super-resolution concepts and techniques for monocular depth estimation and propose a novel CNN-based approach, namely $MSCN_{NS}$, which involves multi-scale sub-pixel convolutions and a neighborhood smoothness constraint. Specifically, $MSCN_{NS}$ makes use of sub-pixel convolutions with multi-scale fusions to retrieve a high-resolution depth map with fine details of the scene. Different from previous multi-scale fusion strategies, those multi-scale features come from supervised scale branches of the network. Furthermore, $MSCN_{NS}$ incorporates a neighborhood smoothness regularization term to make sure that spatially closer pixels with similar features would have close depth values. The effectiveness and efficiency of $MSCN_{NS}$ have been corroborated through extensive experiments conducted on benchmark datasets.

Introduction

Estimating depth from a single (monocular) RGB image is a "holy grail" for low-cost robotics, autonomous navigation, and augmented reality. However, it is an inherently "ill-posed" problem: a single 2D image could technically represent an infinite number of 3D scenes.

While Convolutional Neural Networks (CNNs) have made great strides here, they face a persistent technical hurdle: resolution loss. Standard CNNs downsample images to extract high-level features, then use simple bilinear interpolation or transposed convolutions to "blow the image back up" to its original size. This often results in blurry depth maps that lose fine details and struggle with sharp object boundaries—exactly the details a robot needs to avoid a thin chair leg or a distant obstacle.

The $MSCN_{NS}$ Architecture

The proposed architecture, $MSCN_{NS}$ (Multi-scale Sub-pixel Convolutional Network with Neighborhood Smoothness), treats monocular depth estimation as a super-resolution task. The approach rests on three primary technical pillars:

1. Depth as Super-Resolution

Instead of just upscaling a low-resolution depth prediction, the model formulates the process as a super-resolution task. It utilizes sub-pixel convolutions (also known as "pixel shuffling"), which rearranges channels into spatial pixels. This method is computationally more efficient and preserves significantly more detail than standard upsampling techniques like transposed convolutions.

2. Supervised Multi-Scale Fusion

Most models fuse features from different layers blindly. $MSCN_{NS}$ uses specific "scale branches" that are individually supervised during training. This ensures that the features being fused into the final high-resolution map are actually meaningful at their respective scales. This strategy effectively bridges the gap between early layers (which provide sharp contours) and deep layers (which provide accurate depth values).

Comparison of multi-scale fusion strategies: (a) parallel supervised scale branches used in MSCN vs (b) standard sequential skip connections.

3. Neighborhood Smoothness Constraint

To ensure local consistency, the model integrates a regularization term inspired by Conditional Random Fields (CRFs). This constraint enforces the "prior" that adjacent pixels with similar visual features (color, texture) should likely have similar depth values. Unlike older methods that used hand-crafted features, $MSCN_{NS}$ learns these indicators automatically from the data during end-to-end training.

Experimental Results

The model was trained and evaluated on two gold-standard benchmarks: NYU Depth V2 (indoor) and Make3D (outdoor). The backbone of the system is a modified DenseNet-121, chosen for its efficient feature reuse.

Performance and Speed

On an NVIDIA Titan X, $MSCN_{NS}$ achieves an inference time of just 0.019 seconds per image. For context, previous leading methods took upwards of 0.257 seconds. This 10x speedup makes the model highly viable for real-time robotics applications.

Accuracy and Visual Quality

On the NYU Depth V2 dataset, the model achieved a Relative Error (rel) of 0.128, outperforming several established models. Qualitatively, $MSCN_{NS}$ produces significantly sharper object boundaries and handles fine structures—such as furniture legs or tree branches—much better than predecessors, which often produced "blobby" depth estimates.

Conclusion

By borrowing techniques from the field of image super-resolution, $MSCN_{NS}$ provides a high-fidelity, real-time solution for monocular depth estimation. While monocular methods still face challenges with unusual perspectives or extreme lighting, the authors suggest this multi-scale sub-pixel framework is a general dense prediction tool that could eventually be applied to other tasks like semantic segmentation or optical flow estimation.

Summary updated

Abstract

Depth estimation from a monocular image is of paramount importance in various vision tasks, such as obstacle detection, robot navigation, and 3D reconstruction. However, how to get an accurate depth map with clear details and a fine resolution remains an unresolved issue. As an attempt to solve this problem, we exploit image super-resolution concepts and techniques for monocular depth estimation and propose a novel CNN-based approach, namely <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$MSCN_{NS}$ </tex-math></inline-formula> , which involves multi-scale sub-pixel convolutions and a neighborhood smoothness constraint. Specifically, <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$MSCN_{NS}$ </tex-math></inline-formula> makes use of sub-pixel convolutions with multi-scale fusions to retrieve a high-resolution depth map with fine details of the scene. Different from previous multi-scale fusion strategies, those multi-scale features come from supervised scale branches of the network. Furthermore, <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$MSCN_{NS}$ </tex-math></inline-formula> incorporates a neighborhood smoothness regularization term to make sure that spatially closer pixels with similar features would have close depth values. The effectiveness and efficiency of <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$MSCN_{NS}$ </tex-math></inline-formula> have been corroborated through extensive experiments conducted on benchmark datasets.
BibTeX
@inproceedings{zhao2019super,
  author = {Shiyu Zhao and Lin Zhang and Ying Shen and Shengjie Zhao and Huijuan Zhang},
  title = {{Super-Resolution for Monocular Depth Estimation With Multi-Scale Sub-Pixel Convolutions and a Smoothness Constraint}},
  booktitle = {IEEE Access},
  year = {2019},
  url = {https://openreview.net/forum?id=Ca0UDw98ED},
}
Figure