Abstract
3D shape completion is a critical task in computer vision, involving the reconstruction of full object geometry from partial scans. While view-based methods are effective, they often produce inconsistent results across different perspectives. This paper introduces a Multi-View Consistent Inference framework that optimizes a latent shape descriptor at test time to enforce geometric agreement between views. By treating inference as an energy minimization problem, the method achieves state-of-the-art accuracy on the ShapeNet benchmark without requiring ground-truth data during the inference stage.
Introduction
Predicting the complete 3D geometry of an object from a partial observation—such as a single-view depth image or a sparse point cloud—is a fundamental challenge. Current state-of-the-art "view-based" systems address this by rendering several 2D depth images of the partial object and using neural networks to "fill in" missing regions. However, a significant drawback of these systems is "geometric inconsistency." Because the network processes each view independently, the resulting 3D shape often contains misaligned features or "ghosting" artifacts where different views predict conflicting geometries.
Methodology
The proposed framework addresses inconsistency by shifting from a standard feed-forward inference to an iterative optimization process.
Conditional Generative Network ($G$)
The system utilizes a pre-trained conditional generative network, $G$, consisting of an encoder and a decoder. The encoder maps incomplete depth images ($X_n$) into a latent shape descriptor ($z$). This descriptor is then processed by the decoder to generate completed depth images ($V_n$).
Multi-View Consistent Inference
At test time, the network weights are fixed. Instead of accepting the initial output, the framework iteratively updates the shape descriptor ($z$) to minimize a composite loss function. This function includes a Data Term, which ensures the output remains faithful to the original partial scan, and a Consistency Term, which forces agreement across all rendered views.
The consistency loss is calculated by reprojecting pixels from one view into the camera space of another. To overcome the challenges of discrete pixel grids, the authors implement pseudo-rendering (using a sub-pixel depth buffer) and closest point pooling. This ensures the loss is differentiable and the optimization is stable.
Evaluation and Results
The researchers evaluated the framework using the ShapeNet repository, focusing on eight diverse categories including Airplane, Car, Chair, and Sofa.
Quantitative Results
The method achieved state-of-the-art performance on the ShapeNet benchmark. It reached a Mean Chamfer Distance (CD) of 0.8052, outperforming the previous leading view-based method, MVCN (0.8298), and significantly besting point-based methods like PCN-CD (0.9636).
Qualitative Results
Visually, the optimization process results in significantly cleaner 3D models. By enforcing consistency, the model removes outlier points ("floaters") and correctly aligns thin structural elements, such as chair legs or airplane wings, which are often distorted in independent single-view completions.
Key Insights
- Test-Time Self-Supervision: The framework demonstrates that geometric consistency can act as a self-supervisory signal, allowing for model refinement at inference time without ground-truth labels.
- Manifold Alignment: By optimizing the latent descriptor ($z$) rather than raw pixels, the framework ensures that the resulting shape remains consistent with the distribution of valid objects learned during training.
Limitations and Future Work
The primary limitation is computational cost; the iterative gradient descent (typically 100 steps) makes inference slower than standard one-pass networks, potentially impacting real-time applications. Furthermore, the optimization's success is bounded by the quality of the underlying generative model. Future research may focus on integrating consistency loss directly into the training phase to achieve "consistent-by-design" networks.
Summary updated