Abstract
High-resolution depth maps are essential for various computer vision tasks but are often difficult to obtain directly from consumer-grade sensors. This paper introduces the Joint Convolutional Neural Pyramid (JCNP), a deep learning model designed for joint depth map super-resolution. By framing super-resolution as an image completion task, the JCNP leverages a multi-scale pyramid structure to achieve a significantly larger receptive field than standard convolutional networks. This architecture allows the model to effectively transfer structural information from a high-resolution guidance image while maintaining global consistency and computational efficiency.
Introduction
While high-quality color cameras are ubiquitous, high-resolution (HR) depth sensors remain expensive and limited in availability. Common sensors like the Microsoft Kinect produce low-resolution (LR) depth maps that are often plagued by noise. Joint depth map super-resolution addresses this by using a corresponding HR "guidance" image (e.g., an RGB photo) to reconstruct a high-quality depth map. The primary challenge is accurately transferring sharp edges from the guidance image without introducing "texture bleeding" artifacts. Existing methods often struggle with small receptive fields, limiting their ability to understand global scene structures.
The JCNP Architecture
The JCNP model is composed of three primary sub-networks designed to process and fuse information from the target depth map and the guidance image.
- Target and Guidance CNPs: Two identical Convolutional Neural Pyramids (CNPs) extract features from the LR depth map ($CNP_T$) and the HR guidance image ($CNP_G$).
- Fusion CNN ($CNN_F$): This network concatenates the feature maps from both pyramids. It acts as a filter, determining which structures from the RGB guidance image are relevant for sharpening the depth map.
- Residual Connection: To ensure stable training and better information flow, the original target image is concatenated again at the final stage before the output.
Multi-Level Convolutional Neural Pyramids
The core innovation of the JCNP is the pyramid structure within its feature extraction layers. Unlike "flat" CNNs that maintain a constant spatial resolution, the CNP uses a multi-level approach to expand its context.
As shown in the figure above, each level of the pyramid performs feature extraction, mapping, and reconstruction. By down-sampling features into higher levels and then fusing them back through up-sampling, the network achieves a large receptive field (e.g., 116x116 pixels at 3 levels) without the prohibitive computational costs of a very deep, single-resolution network. This allows the model to suppress noise effectively by considering a wider spatial context.
Experimental Results and Insights
The JCNP was evaluated across several benchmarks, including the Middlebury, NYU v2, and SUN RGB-D datasets.
- Quantitative Performance: The model consistently achieved the lowest Root Mean Square Error (RMSE) across various upscaling factors (2x to 16x). On the NYU v2 dataset at 4x upscaling, JCNP reached an RMSE of 1.59, a significant improvement over previous state-of-the-art methods.
- SR as Image Completion: A key insight of this work is treating super-resolution as an image completion task. Since only a fraction of the HR depth values are known (from the LR input), the large receptive field provided by the pyramid structure is crucial for "filling in" the missing high-frequency details accurately.
- Generalization: Beyond depth maps, the JCNP demonstrated robust performance on other joint-filtering tasks, such as upscaling saliency maps and color-scribble images, proving its versatility as a general-purpose architecture.
Conclusion
The Joint Convolutional Neural Pyramid offers a powerful and efficient solution for depth map super-resolution. By utilizing a multi-scale approach to maximize the receptive field, the JCNP produces sharper, more accurate depth maps than traditional deep learning models. Future work will explore the use of alternative loss functions, such as perceptual or adversarial losses, to further enhance visual sharpness.
Summary updated