Peeking Template Matching for Depth Extension

Simon Korman, Eyal Ofek, Shai Avidan

Introduction

Standard depth sensors, such as the Microsoft Kinect, capture only the visible surfaces of a scene, resulting in "2.5D" data that leaves occluded regions as empty voids. While depth inpainting techniques are effective at filling small holes or reducing noise, "depth extension"—the task of inferring large, entirely unseen 3D volumes—remains a significant challenge. This paper introduces a novel approach to depth extension that leverages self-similarity within a scene to reconstruct complete 3D models from a single-view depth map.

Methodology

The proposed pipeline moves from a partial scan to a complete 3D volume through three primary stages: volume representation, template matching, and global optimization.

Volume Representation with TSDF Bounds

To represent the 3D scene, the authors utilize a Truncated Signed Distance Transform (TSDF). A key innovation is the derivation of mathematical upper ($U$) and lower ($L$) bounds for the TSDF in unknown regions. These bounds define a rigorous "envelope" of possibility, ensuring that synthesized geometry does not violate physical constraints, such as appearing in space already confirmed to be empty "air" by the camera.

Visualization of a distance transform used to represent scene geometry and constraints.

"Peeking" Template Matching

The core of the extension process is a 3D template matcher that identifies repeating structures (e.g., table legs or wall patterns). Because it operates at the boundaries of the visible scene, the matcher must "peek" into unknown space. The authors introduce a scoring function that balances consistency (alignment with known data) and support (the extent of overlap with known data). This prevents the algorithm from making wild guesses based on coincidental matches in sparse data.

Global Optimization via MRF

The template matcher often generates multiple, potentially conflicting hypotheses for the hidden geometry. To resolve these, the scene is modeled as a 3D voxel grid. A Markov Random Field (MRF) is formulated to decide for every voxel whether it is "interior" or "exterior." This optimization, solved via Graph-Cuts, ensures the final surface is smooth and globally consistent with both the original scan and the most reliable hypotheses.

Evaluation and Results

The researchers evaluated their method using the SHREC07 synthetic dataset of CAD models and real-world Kinect scans of complex scenes.

Robustness and Comparison

On the SHREC07 dataset, the algorithm maintained high reconstruction accuracy even when up to 50% of an object's surface was hidden. Qualitatively, the method significantly outperformed standard surface-fitting tools like Poisson Reconstruction. While Poisson interpolation often creates smooth, non-geometric "bubbles" to bridge gaps, this method correctly reconstructed complex, sharp-edged geometry by "copying" it from visible parts of the scene.

Comparison of (a) original RGB, (b) input depth map, (c) a new view showing occlusions, and (d) the final completed 3D reconstruction.

Viewpoint Synthesis

A key capability demonstrated is the generation of novel viewpoints. By completing the 3D model, the system can synthesize depth maps from angles entirely opposite to the original camera position, effectively "seeing" the back of objects from a single-view input.

Conclusion

While highly effective for scenes with repeating structures, the method's primary limitation is its reliance on geometric self-similarity. Unique or asymmetric objects cannot be accurately reconstructed without additional semantic priors. Future work aims to integrate high-level object knowledge to improve performance in non-repetitive environments and optimize the computational cost of high-resolution 3D voxel grid processing.

Summary updated

Abstract

We propose a method that extends a given depth image into regions in 3D that are not visible from the point of view of the camera. The algorithm detects repeated 3D structures in the visible scene and suggests a set of 3D extension hypotheses, which are then combined together through a global 3D MRF discrete optimization. The recovered global 3D surface is consistent with both the input depth map and the hypotheses. A key component of this work is a novel 3D template matcher that is used to detect repeated 3D structure in the scene and to suggest the hypotheses. A unique property of this matcher is that it can handle depth uncertainty. This is crucial because the matcher is required to "peek around the corner", as it operates at the boundaries of the visible 3D scene where depth information is missing. The proposed matcher is fast and is guaranteed to find an approximation to the globally optimal solution. We demonstrate on real-world data that our algorithm is capable of completing a full 3D scene from a single depth image and can synthesize a full depth map from a novel viewpoint of the scene. In addition, we report results on an extensive synthetic set of 3D shapes, which allows us to evaluate the method both qualitatively and quantitatively.
BibTeX
@inproceedings{korman2015peeking,
  author = {Simon Korman and Eyal Ofek and Shai Avidan},
  title = {{Peeking Template Matching for Depth Extension}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2015},
  doi = {10.1109/iccv.2015.251},
}
Figure