Abstract
There is some ambiguity in the 3D shape of an object when the number of observed views is small. Because of this ambiguity, although a 3D object reconstructor can be trained using a single view or a few views per object, reconstructed shapes only fit the observed views and appear incorrect from the unobserved viewpoints. To reconstruct shapes that look reasonable from any viewpoint, we propose to train a discriminator that learns prior knowledge regarding possible views. The discriminator is trained to distinguish the reconstructed views of the observed viewpoints from those of the unobserved viewpoints. The reconstructor is trained to correct unobserved views by fooling the discriminator. Our method outperforms current state-of-the-art methods on both synthetic and natural image datasets; this validates the effectiveness of our method.
Introduction
Reconstructing a 3D object from a single 2D image is an "ill-posed" problem—there are infinitely many 3D shapes that could project to the same 2D silhouette. While models can be trained using 3D ground truth (CAD models), these datasets are expensive and difficult to scale.
The alternative is view-based training, where the model is supervised only by 2D images (silhouettes and textures). However, without 3D guidance, the model often "cheats": it creates a shape that looks perfect from the input viewpoint but resembles a broken, hollow shell or a flat "billboard" from any other angle. This research addresses how to force a model to learn plausible 3D geometry using only 2D observations.
Methodology
The proposed framework consists of a reconstructor (encoder-decoder), a differentiable renderer, and a discriminator.
Reconstructor & Renderer
An input image is processed through a ResNet-18 encoder. Two decoders then generate a 3D mesh (by deforming a template sphere or cube) and a texture map. A differentiable renderer projects this 3D model back into 2D views for supervision.
View Prior Learning (VPL)
This is the core innovation of the work. The authors utilize adversarial training where a discriminator is trained to distinguish between:
- "Real" views: The reconstruction viewed from the original, observed viewpoint (which should match the input image).
- "Fake" views: The reconstruction viewed from a random, unobserved viewpoint.
To fool the discriminator, the reconstructor must ensure that the object looks like a valid member of its class from every angle, effectively learning a 3D prior from 2D data.
Internal Pressure (IP)
To further combat "thin" or hollow reconstructions, the authors introduce a loss that mimics a "visual hull" approach. It applies outward pressure to the mesh vertices, encouraging the model to maximize the volume of the shape within the constraints of the observed silhouettes.
Experiments and Results
Datasets
The method was evaluated on two major benchmarks:
- ShapeNet: A large-scale synthetic dataset of 3D CAD models (~43,000 objects across 13 categories).
- PASCAL (with ImageNet additions): A dataset of real-world natural images with silhouette and viewpoint annotations, testing robustness to noisy data.
Key Results
The proposed method achieved state-of-the-art results for view-based training:
- Quantitative: On the ShapeNet dataset, the VPL method achieved an Intersection over Union (IoU) of 0.513, significantly outperforming the baseline (0.403) and rivaling some methods that use full 3D supervision.
- Qualitative: Reconstructions without VPL often had "missing" backs or strange, thin protrusions. With VPL and Internal Pressure, the models produced full, closed volumes (e.g., the back of a chair or the underside of a car) that looked realistic from all 360 degrees.
- Real-world Performance: On the PASCAL dataset, the method consistently outperformed previous category-specific and category-agnostic models.
Discussion and Conclusion
The most significant insight is that 3D models are not strictly necessary to learn 3D priors. By knowing that an object (e.g., a chair) should look consistent regardless of the camera angle, the discriminator acts as a geometric sanity check. Furthermore, class conditioning and joint texture prediction were found to improve reconstruction accuracy.
Limitations
Despite its success, the method still requires silhouette and viewpoint annotations for training. Future work could involve predicting these end-to-end. Additionally, the model may struggle with complex internal structures or topologies (like a mug handle) due to the constraints of deforming a simple template mesh.
Summary updated