Introduction
Standard depth sensors, such as the Microsoft Kinect, capture only the visible surfaces of a scene, resulting in "2.5D" data that leaves occluded regions as empty voids. While depth inpainting techniques are effective at filling small holes or reducing noise, "depth extension"—the task of inferring large, entirely unseen 3D volumes—remains a significant challenge. This paper introduces a novel approach to depth extension that leverages self-similarity within a scene to reconstruct complete 3D models from a single-view depth map.
Methodology
The proposed pipeline moves from a partial scan to a complete 3D volume through three primary stages: volume representation, template matching, and global optimization.
Volume Representation with TSDF Bounds
To represent the 3D scene, the authors utilize a Truncated Signed Distance Transform (TSDF). A key innovation is the derivation of mathematical upper ($U$) and lower ($L$) bounds for the TSDF in unknown regions. These bounds define a rigorous "envelope" of possibility, ensuring that synthesized geometry does not violate physical constraints, such as appearing in space already confirmed to be empty "air" by the camera.
"Peeking" Template Matching
The core of the extension process is a 3D template matcher that identifies repeating structures (e.g., table legs or wall patterns). Because it operates at the boundaries of the visible scene, the matcher must "peek" into unknown space. The authors introduce a scoring function that balances consistency (alignment with known data) and support (the extent of overlap with known data). This prevents the algorithm from making wild guesses based on coincidental matches in sparse data.
Global Optimization via MRF
The template matcher often generates multiple, potentially conflicting hypotheses for the hidden geometry. To resolve these, the scene is modeled as a 3D voxel grid. A Markov Random Field (MRF) is formulated to decide for every voxel whether it is "interior" or "exterior." This optimization, solved via Graph-Cuts, ensures the final surface is smooth and globally consistent with both the original scan and the most reliable hypotheses.
Evaluation and Results
The researchers evaluated their method using the SHREC07 synthetic dataset of CAD models and real-world Kinect scans of complex scenes.
Robustness and Comparison
On the SHREC07 dataset, the algorithm maintained high reconstruction accuracy even when up to 50% of an object's surface was hidden. Qualitatively, the method significantly outperformed standard surface-fitting tools like Poisson Reconstruction. While Poisson interpolation often creates smooth, non-geometric "bubbles" to bridge gaps, this method correctly reconstructed complex, sharp-edged geometry by "copying" it from visible parts of the scene.
Viewpoint Synthesis
A key capability demonstrated is the generation of novel viewpoints. By completing the 3D model, the system can synthesize depth maps from angles entirely opposite to the original camera position, effectively "seeing" the back of objects from a single-view input.
Conclusion
While highly effective for scenes with repeating structures, the method's primary limitation is its reliance on geometric self-similarity. Unique or asymmetric objects cannot be accurately reconstructed without additional semantic priors. Future work aims to integrate high-level object knowledge to improve performance in non-repetitive environments and optimize the computational cost of high-resolution 3D voxel grid processing.
Summary updated