Abstract
We present a novel approach for completing and reconstructing 3D shapes from incomplete scanned data by using untrained neural networks. Rather than being trained on supervised completion tasks, the network is optimized from scratch on a single testing shape. This allows the model to fully adapt to the specific instance and complete missing data using contextual guidance from known regions. We interpret this "Deep Prior" effect from a Neural Tangent Kernel (NTK) perspective, showing that CNNs naturally reconstruct missing patches based on their proximity in the kernel feature space. This theoretical foundation allows for more efficient network design, enabling the completion of large missing regions with plausible shapes without the need for external training datasets.
The Challenge of 3D Reconstruction
3D scanning is rarely perfect; raw scans often contain "holes" due to occlusion or surface reflectance. Traditionally, these gaps are filled using either hand-crafted geometric rules (like Laplacian smoothing) or supervised deep learning models (like DeepSDF). While geometric rules struggle with complex structures, supervised models require massive datasets and often fail to adapt to unique shapes not present in their training data.
This research bridges the gap by leveraging the Deep Prior—the concept that the architecture of a convolutional neural network (CNN) itself acts as a geometric regularizer. By optimizing a network on a single shape, the model's inductive bias "prefers" the patterns found within that specific object, allowing it to interpolate missing regions naturally.
Methodology: The Unsupervised Pipeline
The core of the approach is a Sparse 3D CNN optimized for every new shape. The shape is represented as a Truncated Signed Distance Function (TSDF) on a 3D grid.
As shown in the pipeline above, the network is fed noise and asked to produce a 3D shape that matches the known parts of the scan. Because of the network's inherent structure, it doesn't just overfit the known pixels; it fills the missing regions with patterns similar to the existing ones. Key architectural choices guided by NTK theory include avoiding skip connections to force the learning of global features and using a multi-scale hierarchy to capture both broad structures and fine details.
The NTK Perspective and Optimization
The authors use NTK theory to prove that CNNs have a "patch-similarity" bias. Essentially, the network's gradients ensure that if two patches are close in the "kernel feature space," they will be reconstructed similarly.
A fascinating insight from this perspective is that networks learn dominant structural features first and noise later. This makes early stopping mandatory; by halting optimization at the right moment, the model avoids replicating scanning artifacts in the completed regions.
Key Results
The method was evaluated using datasets such as ShapeNet Core v2 and the Stanford 3D Scanning Repository.
- Quantitative Performance: The method achieved an F-score of 96.36, significantly higher than traditional Diffusion methods (80.66) and other unsupervised learning methods like Point2Mesh (83.03).
- Qualitative Fidelity: The approach successfully reconstructed complex features, such as thin chair legs and intricate lamp stands, which traditional smoothness-based methods often fail to resolve.
- Adaptability: Unlike supervised models that might force a unique shape into a "generic" template, this method adapts perfectly to the specific style of the input instance.
Limitations and Future Work
Despite its success, the method has limitations. It possesses "semantic blindness," meaning it cannot invent entirely missing components that have no local precedent (e.g., adding legs to a table if all four are missing). Additionally, the computational cost is high, requiring approximately 4 hours of optimization per shape on a GTX 1080 Ti. Future work may explore coordinate-based Implicit Neural Representations to overcome the resolution limits of the current 256³ 3D grid.
Summary updated