Unsupervised Shape Completion via Deep Prior in the Neural Tangent Kernel Perspective

Wenping Wang, Lei Chu, Hao Pan, Wenping Wang 0001

TL;DR

This paper introduces a method to complete missing parts of 3D scans using an untrained neural network optimized on just a single shape. By interpreting the "Deep Prior" effect through the lens of Neural Tangent Kernels (NTK), the authors explain why CNNs naturally fill holes with plausible geometry and use these insights to design a highly effective, unsupervised 3D reconstruction pipeline.

What Problem Does This Paper Address?

3D scanning is rarely perfect. Whether due to occlusion (one part of an object blocking another) or surface reflectance, raw scans often come back with "holes."

Traditionally, researchers filled these holes using two extremes:

  1. Hand-crafted geometric rules: Methods like Laplacian smoothing or symmetry-seeking work well for small holes but fail to capture complex structures.
  2. Supervised Deep Learning: Large models (like DeepSDF) are trained on massive datasets to "guess" missing parts. While powerful, they require huge amounts of data and often struggle to adapt to unique shapes not found in their training set.

The authors bridge this gap by using Deep Prior—the idea that the structure of a convolutional neural network (CNN) itself acts as a "geometric regularizer." You don't need a database of 10,000 chairs if the network's architecture already "prefers" the kind of patterns found in a single chair.

Methods

The core of the approach is a Sparse 3D CNN optimized from scratch for every new shape.

  • Representation: The shape is represented as a Truncated Signed Distance Function (TSDF) on a 3D grid.
  • The Optimization Loop: Instead of training on a dataset, the network is fed random noise and asked to produce a 3D shape that matches the known parts of the scan. Because of the network's inductive bias, it doesn't just "overfit" the known pixels; it naturally interpolates the missing regions with patterns similar to the known ones.
  • The NTK Perspective: The authors use Neural Tangent Kernel (NTK) theory to prove that CNNs have a "patch-similarity" bias. Essentially, the network's gradients ensure that if two patches are close in the "kernel feature space," they will be reconstructed similarly.
  • Architectural Choices: Guided by NTK, they made specific design decisions:
  • No Skip Connections: Unlike standard U-Nets, they avoid skip connections to force the network to learn more global, overlapping features.
  • Multi-scale Hierarchy: They use a three-scale approach to capture both broad structures and fine details.
  • Laplacian Smoothness: A specific loss term is added to ensure local surface regularity.
  • Self-Augmentation: They rotate the input shape 23 different ways during optimization, forcing the network to learn the shape's features from multiple perspectives.

Datasets

The method is unsupervised, so it doesn't "train" on these datasets in the traditional sense. Instead, it uses them for evaluation and testing:

  • ShapeNet Core v2: Specifically chairs and lamps.
  • Stanford 3D Scanning Repository: For free-form, organic shapes (like the famous bunny).
  • Thingi10k: 3D printing models.
  • Free3d: Various CAD models.

Key Insights and Ideas

  • Deep Prior is Kernel Similarity: The "magic" of Deep Prior isn't just luck; it's mathematically grounded in the NTK. The network treats the reconstruction as a kernel regression problem where the "kernel" is defined by the CNN's architecture.
  • Architecture as a Prior: The paper demonstrates that simply choosing a CNN over a standard MLP (Multi-Layer Perceptron) provides a massive boost in completion quality because CNNs are inherently biased toward local spatial correlations.
  • Early Stopping is Mandatory: A fascinating insight from the NTK perspective is that networks learn "dominant" structural features first and "noise" later. By stopping the optimization early (around 2,000 epochs), the model avoids replicating scanning artifacts or noise in the completed regions.

Key Results

The authors compared their method against traditional geometric methods (SPR, VIPSS) and other unsupervised deep learning methods (DGP, Point2Mesh).

  • Quantitative: Their method achieved an F-score of 96.36, significantly higher than traditional Diffusion (80.66) or other unsupervised learning methods like SAL (78.93) and Point2Mesh (83.03).
  • Qualitative: The method successfully reconstructed complex features like chair legs and thin lamp stands that traditional "smoothness-based" methods turned into blobs or ignored entirely.
  • Adaptability: Unlike supervised models (DeepSDF), which might try to turn a unique custom chair into a "generic" chair from its training set, this method adapts perfectly to the specific style of the input shape.

Limitations and Future Work

  • Semantic Blindness: Because the model only knows the single shape it is looking at, it cannot "invent" entirely missing components that have no local precedent. For example, if a table is missing all four legs, the model won't know to add them.
  • Computational Cost: Since the network must be optimized from scratch for every shape, it takes about 4 hours to complete one model on a GTX 1080 Ti. This is much slower than a pre-trained supervised model.
  • Resolution: The use of a 3D grid (TSDF) limits the resolution to the grid size (256³). Future work could explore coordinate-based MLP representations (Implicit Neural Representations) combined with these NTK insights to achieve infinite resolution.

Summary updated

Abstract

We present a novel approach for completing and reconstructing 3D shapes from incomplete scanned data by using deep neural networks. Rather than being trained on supervised completion tasks and applied on a testing shape, the network is optimized from scratch on the single testing shape, to fully adapt to the shape and complete the missing data using contextual guidance from the known regions. The ability to complete missing data by an untrained neural network is usually referred to as the deep prior. In this paper, we interpret the deep prior from a neural tangent kernel (NTK) perspective and show that the completed shape patches by the trained CNN are naturally similar to existing patches, as they are proximate in the kernel feature space induced by NTK. The interpretation allows us to design more efficient network structures and learning mechanisms for the shape completion and reconstruction task. Being more aware of structural regularities than both traditional and other unsupervised learning-based reconstruction methods, our approach completes large missing regions with plausible shapes and complements supervised learning-based methods that use database priors by requiring no extra training data set and showing flexible adaptation to a particular shape instance.
BibTeX
@article{wang2021unsupervised,
  author = {Wenping Wang and Lei Chu and Hao Pan and Wenping Wang},
  title = {{Unsupervised Shape Completion via Deep Prior in the Neural Tangent Kernel Perspective}},
  journal = {ACM Trans. Graphics},
  year = {2021},
  doi = {10.1145/3459234},
}

Loading related content...

Figure