Abstract
This paper introduces VP-MP-GAN, a weakly supervised framework designed to learn high-quality 3D voxel shape distributions using only collections of 2D silhouette images. By integrating a multi-projection GAN architecture with an iteratively trained view-prediction network, the model synthesizes 3D objects from "in-the-wild" images without requiring 3D ground truth, multi-view correspondences, or viewpoint annotations. Experimental results on synthetic and real-world datasets demonstrate that VP-MP-GAN significantly outperforms existing unsupervised methods in both visual quality and quantitative metrics.
Introduction
Creating 3D models is traditionally a data-intensive process. Most deep learning methods for 3D shape generation rely on supervised datasets like ShapeNet, which provide perfect 3D meshes. However, for many real-world categories—such as specific bird species or rare vehicles—3D data is non-existent.
While millions of 2D images are available online, leveraging them for 3D learning presents three major challenges: the lack of 3D ground truth, uncontrolled viewpoints in "in-the-wild" photography, and the absence of multi-view correspondences for individual object instances. The authors address these challenges by learning a 3D distribution that, when projected back to 2D, matches the distribution of a provided image collection.
Methodology
The VP-MP-GAN framework consists of two primary interconnected components that solve the "chicken-and-egg" problem of needing 3D shapes to predict views and view labels to generate 3D shapes.
Multi-projection GAN (MP-GAN)
Unlike standard GANs that utilize a single discriminator, MP-GAN employs multiple discriminators, each assigned to a specific cluster of viewpoints. A 3D generator produces a voxel grid, which is then processed through a differentiable projection layer using ray-casting to create 2D silhouettes. Each discriminator evaluates whether the generated silhouette matches the distribution of real images from its assigned viewpoint.
View Prediction and Clustering
To handle unlabeled training images, a view prediction network ($V$) categorizes images into 16 discrete "view-bins." Because real-world data is often non-uniform (e.g., more photos of car profiles than undercarriages), K-means clustering is used to group these bins into robust view clusters. The system is trained iteratively: the generator improves using current view estimates, and the view predictor is refined using the latest generated 3D shapes.
Experimental Results
The researchers evaluated VP-MP-GAN across four datasets: synthetic chairs (ShapeNet), real-world chairs (Pix3D), real-world cars (Stanford Cars), and real-world birds (CUB-200-2011).
Quantitative and Qualitative Performance
On the synthetic chair dataset, VP-MP-GAN achieved a Fréchet Inception Distance (FID) score of 33.95, significantly outperforming the VAE-based approach (57.38) and the previous state-of-the-art unsupervised method, PrGAN (79.90). Qualitatively, the model produced plausible 3D voxel grids for complex categories like birds and cars, where 3D training data is scarce.
Iterative Learning Success
The iterative training loop proved highly effective. View prediction accuracy began at 42.7% in the initial iteration and rose to 75.6% by the fifth iteration, confirming that the generator and view predictor successfully "teach" each other during the training process.
Discussion and Limitations
VP-MP-GAN demonstrates that multiple discriminators provide clearer gradients to the generator by specializing in specific angles. This makes the model robust to biased viewpoint distributions common in real-world datasets.
However, some limitations remain. Because the model relies on silhouettes, it struggles to reconstruct internal concavities that are not visible in shadow-like projections. Additionally, the current framework assumes unoccluded images where the entire object is visible. Future work may involve incorporating depth information to better capture complex geometries and handling partial occlusions in "in-the-wild" imagery.
Summary updated