StyLitGAN: Image-Based Relighting via Latent Control

Anand Bhattad, James Soole, David A. Forsyth

Abstract

We describe a novel method, StyLitGAN, for relighting and resurfacing images in the absence of labeled data. StyL-itGAN generates images with realistic lighting effects, including cast shadows, soft shadows, inter-reflections, and glossy effects, without the need for paired or CGI data. StyLit-GAN uses an intrinsic image method to decompose an image, followed by a search of the latent space of a pretrained Style-GAN to identify a set of directions. By prompting the model to fix one component (e.g., albedo) and vary another (e.g., shading), we generate relighted images by adding the identi-fied directions to the latent style codes. Quantitative metrics of change in albedo and lighting diversity allow us to choose effective directions using a forward selection process. Qual-itative evaluation confirms the effectiveness of our method.
BibTeX
@inproceedings{bhattad2024stylitgan,
  author = {Anand Bhattad and James Soole and David A. Forsyth},
  title = {{StyLitGAN: Image-Based Relighting via Latent Control}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2024},
  doi = {10.1109/cvpr52733.2024.00405},
}

Loading related content...

Figure