Learning Adversarial 3D Model Generation With 2D Image Enhancer

Jing Zhu, Jin Xie, Yi Fang

Abstract

Recent advancements in generative adversarial nets (GANs) and volumetric convolutional neural networks (CNNs) enable generating 3D models from a probabilistic space. In this paper, we have developed a novel GAN-based deep neural network to obtain a better latent space for the generation of 3D models. In the proposed method, an enhancer neural network is introduced to extract information from other corresponding domains (e.g. image) to improve the performance of the 3D model generator, and the discriminative power of the unsupervised shape features learned from the 3D model discriminator. Specifically, we train the 3D generative adversarial networks on 3D volumetric models, and at the same time, the enhancer network learns image features from rendered images. The evaluations on two large-scale 3D model datasets, ShapeNet and ModelNet, demonstrate that our proposed method can not only generate high-quality 3D models, but also successfully learn discriminative shape representation for classification and retrieval without supervision.

Introduction

Generating realistic 3D models is a significant challenge in computer vision. While Generative Adversarial Networks (GANs) have revolutionized 2D image synthesis, applying them to 3D volumetric data is difficult. Previous attempts, like the seminal 3D-GAN, often produced "broken" models—objects with holes, missing parts, or fragmented surfaces.

Researchers identify two primary culprits for these failures: Data Isolation, where models train only on a single domain (either 3D voxels or 2D projections) and miss cross-domain structural information, and Uninformative Inputs, where standard GANs attempt to map completely random noise vectors directly to complex 3D structures.

Methodology: The Enhancer-Driven 3D GAN

The proposed framework introduces an Enhancer-driven 3D GAN architecture that leverages 2D image features to provide a "structural hint" for 3D synthesis. The system consists of three main components trained jointly:

  1. The 2D Image Enhancer: A 2D GAN (generator $G_E$ and discriminator $D_E$) trained on rendered images of 3D objects to learn high-level visual features.
  2. The 3D Model Generator ($G_M$): Instead of receiving raw random noise, this generator receives high-level features extracted from the 2D Enhancer. This provides a structural foundation that guides the 3D synthesis.
  3. The 3D Model Discriminator ($D_M$): A volumetric CNN that distinguishes between real 3D models and those synthesized by $G_M$, while learning unsupervised shape representations.
The proposed Enhancer-driven 3D GAN framework, showing the joint training of the 2D Enhancer and the 3D Model Generator/Discriminator.

During training, the 2D Enhancer learns what an object looks like from various viewpoints and passes that knowledge to the 3D generator. At inference time, a random vector is fed into the Enhancer, which produces the features needed to drive the 3D generator without requiring an actual input image.

Experimental Results

The researchers utilized ShapeNet for training and ModelNet (10 and 40) to evaluate shape classification and retrieval. The proposed method outperformed existing unsupervised models across several key metrics.

3D Generation Quality

Qualitatively, the generated models are more complete and "solid" compared to the original 3D-GAN. For example, generated tables feature thick, sturdy legs and full tops, whereas previous models often produced fragmented surfaces.

Generated 3D models (64x64x64 resolution) for categories including Airplane, Car, Chair, Table, and Sofa.

Classification and Retrieval

  • Classification Accuracy: On the ModelNet40 benchmark, the method achieved 87.85% accuracy, significantly beating the original 3D-GAN (83.30%) and other unsupervised methods like VConv-DAE (75.50%).
  • Shape Retrieval: In retrieval tasks, the enhancer-based version achieved a Mean Average Precision (MAP) of 65.00% on ModelNet10, compared to 61.82% for the version without the enhancer.

Conclusion

This research demonstrates that 2D images contain vital structural priors that can effectively guide 3D model generation. By forcing the 3D generator to start from 2D image features rather than raw noise, the network learns the "concept" of an object much more effectively. While current models are limited to $64^3$ voxel resolutions, the enhancer concept provides a robust path for improving unsupervised 3D synthesis and feature learning. Future work may extend this modality to include text descriptions or point clouds as additional enhancing domains.

Summary updated

Abstract

Recent advancements in generative adversarial nets (GANs) and volumetric convolutional neural networks (CNNs) enable generating 3D models from a probabilistic space. In this paper, we have developed a novel GAN-based deep neural network to obtain a better latent space for the generation of 3D models. In the proposed method, an enhancer neural network is introduced to extract information from other corresponding domains (e.g. image) to improve the performance of the 3D model generator, and the discriminative power of the unsupervised shape features learned from the 3D model discriminator. Specifically, we train the 3D generative adversarial networks on 3D volumetric models, and at the same time, the enhancer network learns image features from rendered images. Different from the traditional GAN architecture that uses uninformative random vectors as inputs, we feed the high-level image features learned from the enhancer into the 3D model generator for better training. The evaluations on two large-scale 3D model datasets, ShapeNet and ModelNet, demonstrate that our proposed method can not only generate high-quality 3D models, but also successfully learn discriminative shape representation for classification and retrieval without supervision.
BibTeX
@inproceedings{zhu2018learning,
  author = {Jing Zhu and Jin Xie and Yi Fang},
  title = {{Learning Adversarial 3D Model Generation With 2D Image Enhancer}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2018},
  url = {https://openreview.net/forum?id=BkW6xRx_bB},
}
Figure