3D Shape Completion with Multi-View Consistent Inference

Tao Hu, Zhizhong Han, Matthias Zwicker

Abstract

3D shape completion is a critical task in computer vision, involving the reconstruction of full object geometry from partial scans. While view-based methods are effective, they often produce inconsistent results across different perspectives. This paper introduces a Multi-View Consistent Inference framework that optimizes a latent shape descriptor at test time to enforce geometric agreement between views. By treating inference as an energy minimization problem, the method achieves state-of-the-art accuracy on the ShapeNet benchmark without requiring ground-truth data during the inference stage.

Introduction

Predicting the complete 3D geometry of an object from a partial observation—such as a single-view depth image or a sparse point cloud—is a fundamental challenge. Current state-of-the-art "view-based" systems address this by rendering several 2D depth images of the partial object and using neural networks to "fill in" missing regions. However, a significant drawback of these systems is "geometric inconsistency." Because the network processes each view independently, the resulting 3D shape often contains misaligned features or "ghosting" artifacts where different views predict conflicting geometries.

Methodology

The proposed framework addresses inconsistency by shifting from a standard feed-forward inference to an iterative optimization process.

Conditional Generative Network ($G$)

The system utilizes a pre-trained conditional generative network, $G$, consisting of an encoder and a decoder. The encoder maps incomplete depth images ($X_n$) into a latent shape descriptor ($z$). This descriptor is then processed by the decoder to generate completed depth images ($V_n$).

Architecture of the base generative network G, featuring an encoder-decoder structure and shape descriptor z.

Multi-View Consistent Inference

At test time, the network weights are fixed. Instead of accepting the initial output, the framework iteratively updates the shape descriptor ($z$) to minimize a composite loss function. This function includes a Data Term, which ensures the output remains faithful to the original partial scan, and a Consistency Term, which forces agreement across all rendered views.

The inference-time optimization pipeline where the shape descriptor z is iteratively updated to minimize consistency loss across multiple views.

The consistency loss is calculated by reprojecting pixels from one view into the camera space of another. To overcome the challenges of discrete pixel grids, the authors implement pseudo-rendering (using a sub-pixel depth buffer) and closest point pooling. This ensures the loss is differentiable and the optimization is stable.

Evaluation and Results

The researchers evaluated the framework using the ShapeNet repository, focusing on eight diverse categories including Airplane, Car, Chair, and Sofa.

Quantitative Results

The method achieved state-of-the-art performance on the ShapeNet benchmark. It reached a Mean Chamfer Distance (CD) of 0.8052, outperforming the previous leading view-based method, MVCN (0.8298), and significantly besting point-based methods like PCN-CD (0.9636).

Qualitative Results

Visually, the optimization process results in significantly cleaner 3D models. By enforcing consistency, the model removes outlier points ("floaters") and correctly aligns thin structural elements, such as chair legs or airplane wings, which are often distorted in independent single-view completions.

Key Insights

  • Test-Time Self-Supervision: The framework demonstrates that geometric consistency can act as a self-supervisory signal, allowing for model refinement at inference time without ground-truth labels.
  • Manifold Alignment: By optimizing the latent descriptor ($z$) rather than raw pixels, the framework ensures that the resulting shape remains consistent with the distribution of valid objects learned during training.

Limitations and Future Work

The primary limitation is computational cost; the iterative gradient descent (typically 100 steps) makes inference slower than standard one-pass networks, potentially impacting real-time applications. Furthermore, the optimization's success is bounded by the quality of the underlying generative model. Future research may focus on integrating consistency loss directly into the training phase to achieve "consistent-by-design" networks.

Summary updated

Abstract

3D shape completion is important to enable machines to perceive the complete geometry of objects from partial observations. To address this problem, view-based methods have been presented. These methods represent shapes as multiple depth images, which can be back-projected to yield corresponding 3D point clouds, and they perform shape completion by learning to complete each depth image using neural networks. While view-based methods lead to state-of-the-art results, they currently do not enforce geometric consistency among the completed views during the inference stage. To resolve this issue, we propose a multi-view consistent inference technique for 3D shape completion, which we express as an energy minimization problem including a data term and a regularization term. We formulate the regularization term as a consistency loss that encourages geometric consistency among multiple views, while the data term guarantees that the optimized views do not drift away too much from a learned shape descriptor. Experimental results demonstrate that our method completes shapes more accurately than previous techniques.
BibTeX
@inproceedings{hu2020d,
  author = {Tao Hu and Zhizhong Han and Matthias Zwicker},
  title = {{3D Shape Completion with Multi-View Consistent Inference}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2020},
  doi = {10.1609/aaai.v34i07.6734},
}
Figure