Learning View Priors for Single-View 3D Reconstruction

Hiroharu Kato, Tatsuya Harada

Abstract

Reconstructing 3D shapes from a limited number of views often leads to models that only fit the observed angles, appearing distorted from unobserved viewpoints. To address this, we propose View Prior Learning (VPL), which utilizes a discriminator to learn the general appearance of objects from any angle. By training a reconstructor to fool this discriminator, we generate 3D shapes that are realistic from all perspectives. Our method demonstrates state-of-the-art performance on both synthetic (ShapeNet) and natural (PASCAL) image datasets.

Introduction

A fundamental challenge in computer vision is the reconstruction of 3D objects from single 2D images. This task is inherently ill-posed; a single silhouette can correspond to an infinite number of 3D geometries. Traditional view-based training methods often result in "cheating" models that produce high-fidelity reconstructions for the input viewpoint but fail to maintain geometric integrity when rotated.

The View Prior Learning (VPL) Framework

The proposed VPL framework addresses geometric ambiguity by introducing an adversarial training component that teaches the model what objects should look like from unobserved angles.

1. Reconstructor and Differentiable Rendering

The system employs an encoder-decoder architecture to generate a 3D mesh and texture map from a single image. To supervise this process without 3D ground truth, a differentiable renderer projects the mesh back into 2D silhouettes, allowing for direct comparison with the input.

2. Adversarial View Priors

The core innovation is the use of a discriminator trained to distinguish between "real" views (from the observed viewpoint) and "fake" views (from random, unobserved viewpoints). The reconstructor is optimized to produce unobserved views that are indistinguishable from observed ones, effectively learning a category-specific geometric prior.

3. Internal Pressure (IP) Loss

To prevent the mesh from collapsing into flat, 2D-like surfaces, the authors introduced an Internal Pressure (IP) Loss. This loss functions similarly to the "visual hull" concept, encouraging the mesh to expand and occupy a realistic volume.

Experimental Results

The method was evaluated on the ShapeNet synthetic dataset and the PASCAL natural image dataset.

Figure 1: Reconstruction of a 3D chair model from a single 2D image, showing consistent geometry across multiple unobserved viewpoints.

On ShapeNet, the VPL method achieved an Intersection over Union (IoU) of 0.513 for single-view training, a significant improvement over the 0.403 baseline. Qualitatively, the reconstructions are notably more solid and symmetric compared to previous methods, which often produced "broken" or hollow geometries.

Conclusion

By leveraging adversarial learning to establish view priors, this research provides a robust solution to the ambiguity of single-view 3D reconstruction. The ability to generate realistic 3D models from noisy, real-world images marks a significant step toward unsupervised 3D understanding in computer vision.

Summary updated

Abstract

There is some ambiguity in the 3D shape of an object when the number of observed views is small. Because of this ambiguity, although a 3D object reconstructor can be trained using a single view or a few views per object, reconstructed shapes only fit the observed views and appear incorrect from the unobserved viewpoints. To reconstruct shapes that look reasonable from any viewpoint, we propose to train a discriminator that learns prior knowledge regarding possible views. The discriminator is trained to distinguish the reconstructed views of the observed viewpoints from those of the unobserved viewpoints. The reconstructor is trained to correct unobserved views by fooling the discriminator. Our method outperforms current state-of-the-art methods on both synthetic and natural image datasets; this validates the effectiveness of our method.
BibTeX
@inproceedings{kato2019learning,
  author = {Hiroharu Kato and Tatsuya Harada},
  title = {{Learning View Priors for Single-View 3D Reconstruction}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2019},
  url = {https://openreview.net/forum?id=vqAOVOCWIfKW},
}
Figure