Abstract
Recent advancements in generative adversarial nets (GANs) and volumetric convolutional neural networks (CNNs) enable generating 3D models from a probabilistic space. In this paper, we have developed a novel GAN-based deep neural network to obtain a better latent space for the generation of 3D models. In the proposed method, an enhancer neural network is introduced to extract information from other corresponding domains (e.g. image) to improve the performance of the 3D model generator, and the discriminative power of the unsupervised shape features learned from the 3D model discriminator. Specifically, we train the 3D generative adversarial networks on 3D volumetric models, and at the same time, the enhancer network learns image features from rendered images. The evaluations on two large-scale 3D model datasets, ShapeNet and ModelNet, demonstrate that our proposed method can not only generate high-quality 3D models, but also successfully learn discriminative shape representation for classification and retrieval without supervision.
Introduction
Generating realistic 3D models is a significant challenge in computer vision. While Generative Adversarial Networks (GANs) have revolutionized 2D image synthesis, applying them to 3D volumetric data is difficult. Previous attempts, like the seminal 3D-GAN, often produced "broken" models—objects with holes, missing parts, or fragmented surfaces.
Researchers identify two primary culprits for these failures: Data Isolation, where models train only on a single domain (either 3D voxels or 2D projections) and miss cross-domain structural information, and Uninformative Inputs, where standard GANs attempt to map completely random noise vectors directly to complex 3D structures.
Methodology: The Enhancer-Driven 3D GAN
The proposed framework introduces an Enhancer-driven 3D GAN architecture that leverages 2D image features to provide a "structural hint" for 3D synthesis. The system consists of three main components trained jointly:
- The 2D Image Enhancer: A 2D GAN (generator $G_E$ and discriminator $D_E$) trained on rendered images of 3D objects to learn high-level visual features.
- The 3D Model Generator ($G_M$): Instead of receiving raw random noise, this generator receives high-level features extracted from the 2D Enhancer. This provides a structural foundation that guides the 3D synthesis.
- The 3D Model Discriminator ($D_M$): A volumetric CNN that distinguishes between real 3D models and those synthesized by $G_M$, while learning unsupervised shape representations.
During training, the 2D Enhancer learns what an object looks like from various viewpoints and passes that knowledge to the 3D generator. At inference time, a random vector is fed into the Enhancer, which produces the features needed to drive the 3D generator without requiring an actual input image.
Experimental Results
The researchers utilized ShapeNet for training and ModelNet (10 and 40) to evaluate shape classification and retrieval. The proposed method outperformed existing unsupervised models across several key metrics.
3D Generation Quality
Qualitatively, the generated models are more complete and "solid" compared to the original 3D-GAN. For example, generated tables feature thick, sturdy legs and full tops, whereas previous models often produced fragmented surfaces.
Classification and Retrieval
- Classification Accuracy: On the ModelNet40 benchmark, the method achieved 87.85% accuracy, significantly beating the original 3D-GAN (83.30%) and other unsupervised methods like VConv-DAE (75.50%).
- Shape Retrieval: In retrieval tasks, the enhancer-based version achieved a Mean Average Precision (MAP) of 65.00% on ModelNet10, compared to 61.82% for the version without the enhancer.
Conclusion
This research demonstrates that 2D images contain vital structural priors that can effectively guide 3D model generation. By forcing the 3D generator to start from 2D image features rather than raw noise, the network learns the "concept" of an object much more effectively. While current models are limited to $64^3$ voxel resolutions, the enhancer concept provides a robust path for improving unsupervised 3D synthesis and feature learning. Future work may extend this modality to include text descriptions or point clouds as additional enhancing domains.
Summary updated