Abstract
Intrinsic images, in the original sense, are image-like maps of scene properties like depth, normal, albedo or shading. This paper demonstrates that StyleGAN can easily be induced to produce intrinsic images. The procedure is straightforward. We show that, if StyleGAN produces $G(w)$ from latents $w$, then for each type of intrinsic image, there is a fixed offset $d_c$ so that $G(w+d_c)$ is that type of intrinsic image for $G(w)$. Here $d_c$ is independent of $w$. The StyleGAN we used was pretrained by others, so this property is not some accident of our training regime. We show that there are image transformations StyleGAN will not produce in this fashion, so StyleGAN is not a generic image regression engine. It is conceptually exciting that an image generator should “know” and represent intrinsic images. There may also be practical advantages to using a generative model to produce intrinsic images. The intrinsic images obtained from StyleGAN compare well both qualitatively and quantitatively with those obtained by using SOTA image regression techniques; but StyleGAN’s intrinsic images are robust to relighting effects, unlike SOTA methods.
Introduction: Uncovering StyleGAN's Internal World
Does a generative model like StyleGAN simply memorize pixel patterns, or does it develop a functional internal model of the 3D world? Researchers have discovered that StyleGAN doesn't just "paint" pixels; it develops an emergent understanding of 3D geometry and scene physics. By identifying simple, fixed offsets in the latent space, high-quality depth maps, surface normals, and segmentation masks can be extracted from a pre-trained model without any retraining or fine-tuning.
In computer vision, recovering "intrinsic images"—underlying properties like depth, surface orientation (normals), and reflectance (albedo)—is a classic inverse problem. Typically, solving this requires training massive, supervised regression models on specialized datasets. This research explores whether we can "poke" a pre-trained GAN to reveal this hidden knowledge.
Methodology: The Offset Hypothesis
The core of this discovery is a surprisingly simple procedure. Instead of modifying the weights of a pre-trained StyleGAN, the authors explore its latent space ($w^+$).
The researchers hypothesized that for any generated image $G(w)$, there exists a fixed "direction" or offset $d_c$ in the latent space such that $G(w + d_c)$ produces the corresponding intrinsic map for that image. To find these offsets, they use "off-the-shelf" State-of-the-Art (SOTA) predictors (such as ZoeDepth or Omnidata) as temporary teachers. They generate a small set of images, obtain the SOTA prediction, and then optimize for a single vector $d_c$ that minimizes the difference between StyleGAN’s shifted output and the teacher's prediction.
Once the offset $d_c$ is found—a process taking less than two minutes on a modern GPU—it is fixed. It can then be applied to any new latent code $w$ to instantly reveal the intrinsic properties of the generated scene.
Key Insights and Results
Emergent Physics and Linear Latent Spaces
StyleGAN likely learns 3D structure because it is the most efficient way to represent the data. To consistently render a bed from different angles or under varying lighting, the model's internal representations naturally align with physical properties. It is remarkable that a simple additive offset works across thousands of different scenes, suggesting that intrinsic properties are represented as highly organized, directional components within the latent space.
Robustness and Accuracy
One of the most striking findings is that StyleGAN’s internal knowledge is more robust than SOTA regression models. When lighting in a scene changes (relighting), standard depth predictors often get confused by new shadows. StyleGAN’s extracted depth, however, remains consistent because it "knows" the underlying geometry hasn't changed.
In quantitative tests, StyleGAN’s extracted depth maps outperformed some SOTA calibration methods, achieving an L1 error of 0.1019 compared to Omnidata-v2's 0.1237. Furthermore, StyleGAN exhibits high precision in segmentation, identifying multiple objects (like lamps) that SOTA segmenters often miss, likely because the model itself "decided" to place them there during generation.
Limitations and Future Outlook
Currently, this method works best on images generated by StyleGAN. Applying this to real-world photographs requires "inverting" the photo into StyleGAN’s latent space, a process that is not yet perfect. Additionally, while global structure is excellent, StyleGAN-generated normals can sometimes be less sharp than those from supervised SOTA models.
Despite these limitations, the discovery that generative models inherently represent scene intrinsics opens a promising path for future research. This property likely exists in other architectures, such as Diffusion Models, suggesting a new paradigm for "extracting" vision models from generative ones.
Source: StyleGAN knows Normal, Depth, Albedo, and More, Bhattad et al. (2023).
Summary updated