Abstract
3D shape completion is important to enable machines to perceive the complete geometry of objects from partial observations. To address this problem, view-based methods have been presented. These methods represent shapes as multiple depth images, which can be back-projected to yield corresponding 3D point clouds, and they perform shape completion by learning to complete each depth image using neural networks. While view-based methods lead to state-of-the-art results, they currently do not enforce geometric consistency among the completed views during the inference stage. To resolve this issue, we propose a multi-view consistent inference technique for 3D shape completion, which we express as an energy minimization problem including a data term and a regularization term. We formulate the regularization term as a consistency loss that encourages geometric consistency among multiple views, while the data term guarantees that the optimized views do not drift away too much from a learned shape descriptor. Experimental results demonstrate that our method completes shapes more accurately than previous techniques.
Introduction
Predicting a full 3D model from a partial observation, such as a single depth scan, remains a fundamental challenge in computer vision. A popular strategy is "view-based" reconstruction, where a system predicts multiple 2D depth images from various angles and fuses them into a 3D shape. While 2D views are computationally efficient for neural networks, they often lack geometric consistency at inference time. Without ground truth data to enforce alignment, a predicted feature in one view may not correspond to the same surface in another, leading to noisy and inconsistent 3D outputs.
Methodology
To address this, we propose a multi-view consistent inference technique that treats shape completion as an energy minimization problem during the inference stage. Rather than relying solely on the initial output of a pre-trained network, the system optimizes the representation to ensure global coherence.
The framework consists of three primary components:
- Conditional Generative Net ($G$): A pre-trained encoder-decoder architecture. It processes incomplete depth images ($X$) and a latent shape descriptor ($z$) to generate completed views ($V$).
- Consistency Loss ($L_{con}$): This term enforces geometric agreement between views. For any pair of views, pixels from a source view are reprojected into the coordinate space of a target view. The distance between these reprojected points and the target surface is minimized.
- Generator Loss ($L_{gen}$): This data term acts as a regularizer, ensuring the optimized shape descriptor ($z$) remains close to the network's initial prediction, preventing the generation of unrealistic geometries.
During inference, the network's weights remain fixed. The system performs iterative gradient descent (typically 100 steps) to optimize only the shape descriptor $z$, minimizing the total energy.
Differentiable Consistency Calculation
A significant challenge in view-based consistency is the discrete nature of depth pixels. To overcome this, the method employs a "pseudo-rendering" technique to make the optimization process stable and differentiable.
As shown in the figure above, the process involves back-projecting a source view $V_s$ into a 3D point cloud and then re-projecting it into a target view $V_t$. By using a sub-pixel depth buffer and "closest point pooling," the consistency loss becomes differentiable and robust to minor alignment errors or the "collision effect" where multiple points project to the same pixel.
Experimental Results
The proposed method was evaluated on the ShapeNet dataset across eight categories, including Airplane, Car, and Sofa. It was compared against state-of-the-art benchmarks such as Point Completion Network (PCN) and Multi-view Completion Network (MVCN).
Quantitatively, the method achieved the lowest Mean Chamfer Distance (CD) across nearly all categories. On the ShapeNet test set, it reached an average CD of 0.8052, significantly outperforming MVCN (0.8298) and point-based methods like PCN-CD (0.9636). Qualitatively, the optimization results in smoother surfaces, better-defined thin structures (e.g., chair legs), and fewer floating artifacts compared to non-optimized baselines.
Discussion and Limitations
The core insight of this work is that a pre-trained model's output can be significantly improved by using geometric constraints as a self-supervision signal at test time. By optimizing the latent shape descriptor rather than raw pixels, the model is constrained to stay within the manifold of realistic, learned shapes.
However, the method introduces computational overhead due to the iterative optimization required at inference time, which may limit its applicability in real-time scenarios. Furthermore, the final quality is bounded by the representational capacity of the underlying generative model. Future work may involve integrating this consistency loss directly into the training phase or developing more efficient optimization algorithms.
Summary updated