IntrinsicDiffusion: Joint Intrinsic Layers from Latent Diffusion Models
Lists
Only you see this
Abstract
Reasoning about the intrinsic properties of an image, such as albedo, illumination, and surface geometry, is a long-standing problem with many applications in image editing and compositing. Existing solutions to this ill-posed problem either heavily rely on manually designed priors or learn priors from limited datasets that lack diversity. Hence, they fall short in generalizing to in-the-wild test scenarios. In this paper, we show that a large-scale text-to-image generation model trained on a massive amount of visual data can implicitly learn intrinsic image priors. In particular, we introduce a novel conditioning mechanism built on top of a pre-trained foundational image generation model to jointly predict multiple intrinsic modalities from an input image. We demonstrate that predicting different modalities in a collaborative manner improves the overall quality. This design also enables mixing datasets with annotations of only a subset of the modalities during training, contributing to the generalizability of our approach. Our method achieves state-of-the-art performance in intrinsic image decomposition, both qualitatively and quantitatively. We also demonstrate downstream image editing applications, such as relighting and retexturing.
BibTeX
@inproceedings{luo2024intrinsicdiffusion,
author = {Jundan Luo and Duygu Ceylan and Jae Shin Yoon and Nanxuan Zhao and Julien Philip and Anna Frühstück and Wenbin Li and Christian Richardt and Tuanfeng Yang Wang},
title = {{IntrinsicDiffusion: Joint Intrinsic Layers from Latent Diffusion Models}},
booktitle = {SIGGRAPH (Conference Paper Track)},
year = {2024},
doi = {10.1145/3641519.3657472},
}Loading related content...