Joint convolutional neural pyramid for depth map super-resolution

Yi Xiao, Xiang Cao, Xianyi Zhu, Renzhi Yang, Yan Zheng

Abstract

High-resolution depth maps are essential for various computer vision tasks but are often difficult to obtain directly from consumer-grade sensors. This paper introduces the Joint Convolutional Neural Pyramid (JCNP), a deep learning model designed for joint depth map super-resolution. By framing super-resolution as an image completion task, the JCNP leverages a multi-scale pyramid structure to achieve a significantly larger receptive field than standard convolutional networks. This architecture allows the model to effectively transfer structural information from a high-resolution guidance image while maintaining global consistency and computational efficiency.

Introduction

While high-quality color cameras are ubiquitous, high-resolution (HR) depth sensors remain expensive and limited in availability. Common sensors like the Microsoft Kinect produce low-resolution (LR) depth maps that are often plagued by noise. Joint depth map super-resolution addresses this by using a corresponding HR "guidance" image (e.g., an RGB photo) to reconstruct a high-quality depth map. The primary challenge is accurately transferring sharp edges from the guidance image without introducing "texture bleeding" artifacts. Existing methods often struggle with small receptive fields, limiting their ability to understand global scene structures.

The JCNP Architecture

The JCNP model is composed of three primary sub-networks designed to process and fuse information from the target depth map and the guidance image.

The overall JCNP architecture, showing the parallel CNP networks for target and guidance images followed by a fusion CNN.
  1. Target and Guidance CNPs: Two identical Convolutional Neural Pyramids (CNPs) extract features from the LR depth map ($CNP_T$) and the HR guidance image ($CNP_G$).
  2. Fusion CNN ($CNN_F$): This network concatenates the feature maps from both pyramids. It acts as a filter, determining which structures from the RGB guidance image are relevant for sharpening the depth map.
  3. Residual Connection: To ensure stable training and better information flow, the original target image is concatenated again at the final stage before the output.

Multi-Level Convolutional Neural Pyramids

The core innovation of the JCNP is the pyramid structure within its feature extraction layers. Unlike "flat" CNNs that maintain a constant spatial resolution, the CNP uses a multi-level approach to expand its context.

The internal structure of a CNP, illustrating the down-sampling and up-sampling paths across multiple levels to expand the receptive field.

As shown in the figure above, each level of the pyramid performs feature extraction, mapping, and reconstruction. By down-sampling features into higher levels and then fusing them back through up-sampling, the network achieves a large receptive field (e.g., 116x116 pixels at 3 levels) without the prohibitive computational costs of a very deep, single-resolution network. This allows the model to suppress noise effectively by considering a wider spatial context.

Experimental Results and Insights

The JCNP was evaluated across several benchmarks, including the Middlebury, NYU v2, and SUN RGB-D datasets.

  • Quantitative Performance: The model consistently achieved the lowest Root Mean Square Error (RMSE) across various upscaling factors (2x to 16x). On the NYU v2 dataset at 4x upscaling, JCNP reached an RMSE of 1.59, a significant improvement over previous state-of-the-art methods.
  • SR as Image Completion: A key insight of this work is treating super-resolution as an image completion task. Since only a fraction of the HR depth values are known (from the LR input), the large receptive field provided by the pyramid structure is crucial for "filling in" the missing high-frequency details accurately.
  • Generalization: Beyond depth maps, the JCNP demonstrated robust performance on other joint-filtering tasks, such as upscaling saliency maps and color-scribble images, proving its versatility as a general-purpose architecture.

Conclusion

The Joint Convolutional Neural Pyramid offers a powerful and efficient solution for depth map super-resolution. By utilizing a multi-scale approach to maximize the receptive field, the JCNP produces sharper, more accurate depth maps than traditional deep learning models. Future work will explore the use of alternative loss functions, such as perceptual or adversarial losses, to further enhance visual sharpness.

Summary updated

Abstract

High-resolution depth map can be inferred from a low-resolution one with the guidance of an additional high-resolution texture map of the same scene. Recently, deep neural networks with large receptive fields are shown to benefit applications such as image completion. Our insight is that super resolution is similar to image completion, where only parts of the depth values are precisely known. In this paper, we present a joint convolutional neural pyramid model with large receptive fields for joint depth map super-resolution. Our model consists of three sub-networks, two convolutional neural pyramids concatenated by a normal convolutional neural network. The convolutional neural pyramids extract information from large receptive fields of the depth map and guidance map, while the convolutional neural network effectively transfers useful structures of the guidance image to the depth image. Experimental results show that our model outperforms existing state-of-the-art algorithms not only on data pairs of RGB/depth images, but also on other data pairs like color/saliency and color-scribbles/colorized images.
BibTeX
@misc{xiao2018joint,
  author = {Yi Xiao and Xiang Cao and Xianyi Zhu and Renzhi Yang and Yan Zheng},
  title = {{Joint convolutional neural pyramid for depth map super-resolution}},
  year = {2018},
  eprint = {1801.00968},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1801.00968},
}
Figure