Image GANs meet Differentiable Rendering for Inverse Graphics and Interpretable 3D Neural Rendering

Yuxuan Zhang, Wenzheng Chen, Huan Ling, Jun Gao, Yinan Zhang, Antonio Torralba, Sanja Fidler

Introduction

Differentiable rendering has paved the way for training neural networks to perform "inverse graphics" tasks, such as predicting 3D geometry from monocular photographs. However, training high-performing models typically requires multi-view imagery or expensive 3D ground truth, which are not readily available in practice.

This paper introduces a clever workaround: treating a pre-trained StyleGAN as a multi-view data generator. By training a network to reconstruct 3D shapes from these generated images, the authors use that acquired knowledge to "unlock" explicit 3D controls—like camera angle and shape—within the GAN itself.

The Problem: Data Bottlenecks in Inverse Graphics

In computer vision, inverse graphics is the task of taking a flat 2D image and deducing the 3D geometry, texture, and lighting that created it. Traditionally, training these models requires either:

  1. 3D Ground Truth: Highly accurate 3D models (like ShapeNet), which often look synthetic and fail to translate well to real-world photos.
  2. Multi-view Imagery: Multiple photos of the exact same object from different angles, which are incredibly difficult to collect at scale for real-world categories like cars or birds.

Meanwhile, Generative Adversarial Networks (GANs) like StyleGAN have become masters at creating hyper-realistic images. Researchers have noticed that GANs seem to learn 3D structure implicitly—tweaking certain numbers in the GAN's "latent code" can make a generated car appear to rotate. However, these codes are "black boxes" lacking specific, interpretable controls for physical properties.

Methods: Linking Neural and Differentiable Renderers

The authors propose a framework that links a GAN (the neural renderer) and a Differentiable Renderer (a math-based tool that turns 3D meshes into 2D images).

Overview of the proposed framework linking a StyleGAN data generator with an Inverse Graphics Network.

The pipeline operates in several key steps:

  1. GAN as a Data Factory: The authors discovered that specific layers of StyleGAN control viewpoint. By fixing these "viewpoint codes" and varying the rest, they can generate thousands of images of different objects from the exact same angle. This creates a massive, synthetic multi-view dataset of realistic-looking objects.
Examples of synthetic multi-view datasets generated by manipulating StyleGAN latent codes.
  1. Training the Inverse Graphics Network (IGN): A model is trained to predict a 3D mesh and texture from these GAN images. Because the system knows which images represent the "same" object from different angles, it uses a multi-view consistency loss. This forces the network to predict matching 3D shapes regardless of whether it's looking at the front or side of a car.
  2. Disentangling the GAN: Once trained, the IGN acts as a teacher. It analyzes GAN-generated images and maps specific parts of the GAN's latent code to explicit 3D properties (mesh, texture, lighting). This results in StyleGAN-R, a disentangled version of the GAN where 3D properties can be explicitly controlled.
  3. Iterative Refinement: The entire architecture is fine-tuned iteratively using cycle-consistency losses, ensuring the 3D predictions and the GAN's internal logic are perfectly aligned.

Key Insights and Results

  • The "1-Minute" Annotation: To initialize the system, a human only needs to spend about one minute labeling a few GAN viewpoints with rough angles (e.g., "front-left view"). The differentiable renderer then automatically fine-tunes these camera positions during training. This requires 10,000x less human annotation effort than traditional datasets like Pascal3D (1 minute vs. 200+ hours).
  • GANs as "Free" Multi-view Data: A high-quality GAN has already distilled the 3D structure of the world from millions of single photos. This method successfully extracts that implicit knowledge without needing to capture real-world multi-view datasets.
  • Neural vs. Traditional Rendering: Traditional renderers struggle with complex effects like reflections or soft shadows. By using StyleGAN-R as a "neural renderer," the authors produce 3D-controllable images that retain the photorealistic "eye candy" of a standard GAN.
  • Performance: In user studies, participants preferred the 3D reconstructions from this method 57.5% of the time, compared to only 25.9% for the previous state-of-the-art baseline. Furthermore, the authors demonstrated the ability to take a real photo, invert it into the GAN, and swap its shape or texture while keeping the background consistent.

Limitations and Future Work

While highly effective, the approach has limitations. The model relies on a simple lighting model (Spherical Harmonics) that sometimes fails to fully separate complex lighting from an object's intrinsic texture. Additionally, while it excels with rigid objects like cars, it struggles with articulated subjects like horses and birds, as the GAN doesn't always keep limb positions consistent across different viewpoints. Finally, the system struggles with out-of-distribution images (e.g., highly unusual car shapes) because the "teacher" GAN cannot accurately represent them. Future work will likely focus on more complex lighting models and better handling of non-rigid, moving parts.

Summary updated

Abstract

Differentiable rendering has paved the way to training neural networks to perform “inverse graphics” tasks such as predicting 3D geometry from monocular photographs. To train high performing models, most of the current approaches rely on multi-view imagery which are not readily available in practice. Recent Generative Adversarial Networks (GANs) that synthesize images, in contrast, seem to acquire 3D knowledge implicitly during training: object viewpoints can be manipulated by simply manipulating the latent codes. However, these latent codes often lack further physical interpretation and thus GANs cannot easily be inverted to perform explicit 3D reasoning. In this paper, we aim to extract and disentangle 3D knowledge learned by generative models by utilizing differentiable renderers. Key to our approach is to exploit GANs as a multi-view data generator to train an inverse graphics network using an off-the-shelf differentiable renderer, and the trained inverse graphics network as a teacher to disentangle the GAN's latent code into interpretable 3D properties. The entire architecture is trained iteratively using cycle consistency losses. We show that our approach significantly outperforms state-of-the-art inverse graphics networks trained on existing datasets, both quantitatively and via user studies. We further showcase the disentangled GAN as a controllable 3D “neural renderer", complementing traditional graphics renderers.
BibTeX
@inproceedings{zhang2021image,
  author = {Yuxuan Zhang and Wenzheng Chen and Huan Ling and Jun Gao and Yinan Zhang and Antonio Torralba and Sanja Fidler},
  title = {{Image GANs meet Differentiable Rendering for Inverse Graphics and Interpretable 3D Neural Rendering}},
  booktitle = {International Conference on Learning Representations},
  year = {2021},
  url = {https://openreview.net/forum?id=yWkP7JuHX1},
}
Figure