StyleGAN knows Normal, Depth, Albedo, and More

Anand Bhattad, Daniel McKee, Derek Hoiem, David A. Forsyth

Abstract

Intrinsic images, in the original sense, are image-like maps of scene properties like depth, normal, albedo or shading. This paper demonstrates that StyleGAN can easily be induced to produce intrinsic images. The procedure is straightforward. We show that, if StyleGAN produces $G(w)$ from latents $w$, then for each type of intrinsic image, there is a fixed offset $d_c$ so that $G(w+d_c)$ is that type of intrinsic image for $G(w)$. Here $d_c$ is independent of $w$. The StyleGAN we used was pretrained by others, so this property is not some accident of our training regime. We show that there are image transformations StyleGAN will not produce in this fashion, so StyleGAN is not a generic image regression engine. It is conceptually exciting that an image generator should “know” and represent intrinsic images. There may also be practical advantages to using a generative model to produce intrinsic images. The intrinsic images obtained from StyleGAN compare well both qualitatively and quantitatively with those obtained by using SOTA image regression techniques; but StyleGAN’s intrinsic images are robust to relighting effects, unlike SOTA methods.

Introduction: Uncovering StyleGAN's Internal World

Does a generative model like StyleGAN simply memorize pixel patterns, or does it develop a functional internal model of the 3D world? Researchers have discovered that StyleGAN doesn't just "paint" pixels; it develops an emergent understanding of 3D geometry and scene physics. By identifying simple, fixed offsets in the latent space, high-quality depth maps, surface normals, and segmentation masks can be extracted from a pre-trained model without any retraining or fine-tuning.

In computer vision, recovering "intrinsic images"—underlying properties like depth, surface orientation (normals), and reflectance (albedo)—is a classic inverse problem. Typically, solving this requires training massive, supervised regression models on specialized datasets. This research explores whether we can "poke" a pre-trained GAN to reveal this hidden knowledge.

Methodology: The Offset Hypothesis

The core of this discovery is a surprisingly simple procedure. Instead of modifying the weights of a pre-trained StyleGAN, the authors explore its latent space ($w^+$).

Intrinsic images (normals, depth, albedo, shading, and segmentation) extracted from StyleGAN using fixed latent offsets.

The researchers hypothesized that for any generated image $G(w)$, there exists a fixed "direction" or offset $d_c$ in the latent space such that $G(w + d_c)$ produces the corresponding intrinsic map for that image. To find these offsets, they use "off-the-shelf" State-of-the-Art (SOTA) predictors (such as ZoeDepth or Omnidata) as temporary teachers. They generate a small set of images, obtain the SOTA prediction, and then optimize for a single vector $d_c$ that minimizes the difference between StyleGAN’s shifted output and the teacher's prediction.

Once the offset $d_c$ is found—a process taking less than two minutes on a modern GPU—it is fixed. It can then be applied to any new latent code $w$ to instantly reveal the intrinsic properties of the generated scene.

Key Insights and Results

Emergent Physics and Linear Latent Spaces

StyleGAN likely learns 3D structure because it is the most efficient way to represent the data. To consistently render a bed from different angles or under varying lighting, the model's internal representations naturally align with physical properties. It is remarkable that a simple additive offset works across thousands of different scenes, suggesting that intrinsic properties are represented as highly organized, directional components within the latent space.

Robustness and Accuracy

One of the most striking findings is that StyleGAN’s internal knowledge is more robust than SOTA regression models. When lighting in a scene changes (relighting), standard depth predictors often get confused by new shadows. StyleGAN’s extracted depth, however, remains consistent because it "knows" the underlying geometry hasn't changed.

Comparison of various intrinsic maps generated for a bedroom scene, demonstrating the model's ability to segment objects and estimate geometry.

In quantitative tests, StyleGAN’s extracted depth maps outperformed some SOTA calibration methods, achieving an L1 error of 0.1019 compared to Omnidata-v2's 0.1237. Furthermore, StyleGAN exhibits high precision in segmentation, identifying multiple objects (like lamps) that SOTA segmenters often miss, likely because the model itself "decided" to place them there during generation.

Limitations and Future Outlook

Currently, this method works best on images generated by StyleGAN. Applying this to real-world photographs requires "inverting" the photo into StyleGAN’s latent space, a process that is not yet perfect. Additionally, while global structure is excellent, StyleGAN-generated normals can sometimes be less sharp than those from supervised SOTA models.

Despite these limitations, the discovery that generative models inherently represent scene intrinsics opens a promising path for future research. This property likely exists in other architectures, such as Diffusion Models, suggesting a new paradigm for "extracting" vision models from generative ones.

Source: StyleGAN knows Normal, Depth, Albedo, and More, Bhattad et al. (2023).

Summary updated

Abstract

Intrinsic images, in the original sense, are image-like maps of scene properties like depth, normal, albedo or shading. This paper demonstrates that StyleGAN can easily be induced to produce intrinsic images. The procedure is straightforward. We show that, if StyleGAN produces $G({w})$ from latents ${w}$, then for each type of intrinsic image, there is a fixed offset ${d}_c$ so that $G({w}+{d}_c)$ is that type of intrinsic image for $G({w})$. Here ${d}_c$ is {\em independent of ${w}$}. The StyleGAN we used was pretrained by others, so this property is not some accident of our training regime. We show that there are image transformations StyleGAN will {\em not} produce in this fashion, so StyleGAN is not a generic image regression engine. It is conceptually exciting that an image generator should ``know'' and represent intrinsic images. There may also be practical advantages to using a generative model to produce intrinsic images. The intrinsic images obtained from StyleGAN compare well both qualitatively and quantitatively with those obtained by using SOTA image regression techniques; but StyleGAN's intrinsic images are robust to relighting effects, unlike SOTA methods.
BibTeX
@inproceedings{bhattad2023stylegan,
  author = {Anand Bhattad and Daniel McKee and Derek Hoiem and David A. Forsyth},
  title = {{StyleGAN knows Normal, Depth, Albedo, and More}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2023},
  url = {https://openreview.net/forum?id=FYqqvQdXhZ},
}
Figure