Abstract
Reconstructing 3D shapes from a limited number of views often leads to models that only fit the observed angles, appearing distorted from unobserved viewpoints. To address this, we propose View Prior Learning (VPL), which utilizes a discriminator to learn the general appearance of objects from any angle. By training a reconstructor to fool this discriminator, we generate 3D shapes that are realistic from all perspectives. Our method demonstrates state-of-the-art performance on both synthetic (ShapeNet) and natural (PASCAL) image datasets.
Introduction
A fundamental challenge in computer vision is the reconstruction of 3D objects from single 2D images. This task is inherently ill-posed; a single silhouette can correspond to an infinite number of 3D geometries. Traditional view-based training methods often result in "cheating" models that produce high-fidelity reconstructions for the input viewpoint but fail to maintain geometric integrity when rotated.
The View Prior Learning (VPL) Framework
The proposed VPL framework addresses geometric ambiguity by introducing an adversarial training component that teaches the model what objects should look like from unobserved angles.
1. Reconstructor and Differentiable Rendering
The system employs an encoder-decoder architecture to generate a 3D mesh and texture map from a single image. To supervise this process without 3D ground truth, a differentiable renderer projects the mesh back into 2D silhouettes, allowing for direct comparison with the input.
2. Adversarial View Priors
The core innovation is the use of a discriminator trained to distinguish between "real" views (from the observed viewpoint) and "fake" views (from random, unobserved viewpoints). The reconstructor is optimized to produce unobserved views that are indistinguishable from observed ones, effectively learning a category-specific geometric prior.
3. Internal Pressure (IP) Loss
To prevent the mesh from collapsing into flat, 2D-like surfaces, the authors introduced an Internal Pressure (IP) Loss. This loss functions similarly to the "visual hull" concept, encouraging the mesh to expand and occupy a realistic volume.
Experimental Results
The method was evaluated on the ShapeNet synthetic dataset and the PASCAL natural image dataset.
On ShapeNet, the VPL method achieved an Intersection over Union (IoU) of 0.513 for single-view training, a significant improvement over the 0.403 baseline. Qualitatively, the reconstructions are notably more solid and symmetric compared to previous methods, which often produced "broken" or hollow geometries.
Conclusion
By leveraging adversarial learning to establish view priors, this research provides a robust solution to the ambiguity of single-view 3D reconstruction. The ability to generate realistic 3D models from noisy, real-world images marks a significant step toward unsupervised 3D understanding in computer vision.
Summary updated