Abstract
Differentiable rendering has paved the way to training neural networks to perform "inverse graphics" tasks such as predicting 3D geometry from monocular photographs. To train high performing models, most of the current approaches rely on multi-view imagery which are not readily available in practice. Recent Generative Adversarial Networks (GANs) that synthesize images, in contrast, seem to acquire 3D knowledge implicitly during training: object viewpoints can be manipulated by simply manipulating the latent codes. However, these latent codes often lack further physical interpretation and thus GANs cannot easily be inverted to perform explicit 3D reasoning. In this paper, we aim to extract and disentangle 3D knowledge learned by generative models by utilizing differentiable renderers. Key to our approach is to exploit GANs as a multi-view data generator to train an inverse graphics network using an off-the-shelf differentiable renderer, and the trained inverse graphics network as a teacher to disentangle the GAN's latent code into interpretable 3D properties. The entire architecture is trained iteratively using cycle consistency losses. We show that our approach significantly outperforms state-of-the-art inverse graphics networks trained on existing datasets, both quantitatively and via user studies. We further showcase the disentangled GAN as a controllable 3D "neural renderer", complementing traditional graphics renderers.
Introduction
Inferring 3D properties—such as geometry, texture, and lighting—from a single 2D image is a cornerstone of computer vision known as inverse graphics. Traditionally, training these models requires either massive 3D ground-truth datasets or multi-view imagery, both of which are difficult to obtain for real-world objects.
Modern Generative Adversarial Networks (GANs), like StyleGAN, have shown an incredible ability to generate photorealistic images and appear to learn 3D structures implicitly. However, their latent spaces are "black boxes" lacking physical interpretability. This research introduces a framework to bridge the gap between the photorealism of GANs and the physical interpretability of differentiable rendering, transforming a standard GAN into a controllable 3D neural renderer.
Methodology
The proposed framework utilizes a three-stage pipeline that marries StyleGAN with a differentiable graphics renderer (DIB-R).
- GAN as a Data Generator: The authors exploit the fact that StyleGAN layers control different levels of detail. By fixing "content" codes and varying "viewpoint" codes, they generate synthetic multi-view datasets. This requires only minimal human effort (~1 minute) to categorize viewpoints into coarse bins.
- Training the Inverse Graphics Network (IGN): Using the GAN-generated multi-view data, the authors train an IGN. The availability of multiple views of the same "virtual" object allows for a multi-view consistency loss, forcing the network to predict a single, coherent 3D mesh that remains accurate from all generated angles.
- Disentangling the GAN (StyleGAN-R): The trained IGN then acts as a "teacher" for the GAN. A mapping network is built to translate explicit 3D parameters (mesh, texture, camera) into StyleGAN’s latent space.
Interpretable 3D Neural Rendering
By fine-tuning the system with cycle-consistency losses, the authors create StyleGAN-R. This model ensures that if 3D properties are predicted from a GAN image and then "re-rendered" through the GAN, the resulting image matches the original.
This architecture allows for sophisticated image manipulation. For instance, one can take a real photo of a car, extract its 3D properties, and swap its body shape with another model while maintaining the original texture and background.
Key Results and Insights
The model was evaluated across three categories: Cars (LSUN Car), Horses (LSUN Horse), and Birds (NABirds).
- Superior Reconstruction: On the Pascal3D+ benchmark, the model significantly outperformed baselines. In user studies, participants preferred the authors' 3D reconstructions 57.5% of the time.
- Unsupervised 3D Engines: The research demonstrates that GANs trained on 2D images are essentially unsupervised 3D engines. We do not need to build 3D datasets from scratch; we simply need to label the knowledge already latent within the GAN.
- Generalization: Despite being trained on "fake" GAN-generated images, the model generalizes remarkably well to real-world photographs and web imagery.
Limitations and Future Work
While successful, the model currently uses a simplified lighting model that can sometimes "bake" reflections into textures. It is also limited by the GAN's learned distribution; it cannot reconstruct shapes the GAN has never seen (e.g., a highly unique custom car). Future work aims to incorporate more complex lighting and handle articulated objects, such as animals with moving joints, more explicitly.
Summary updated