Unsupervised Single-Image Intrinsic Image Decomposition with LiDAR Intensity Enhanced Training

Shogo Sato, Takuhiro Kaneko, Kazuhiko Murasaki, Taiga Yoshida, Ryuichi Tanida, Akisato KimuraWACV 2025

TL;DR

This paper introduces LIET, a novel unsupervised method for intrinsic image decomposition (IID) that can infer albedo and shade from a single image during inference. It achieves this by leveraging LiDAR intensity during training to enhance the decomposition quality, particularly in reducing shadows, while maintaining the practical advantage of single-image inference.

What Problem Does This Paper Address?

Intrinsic Image Decomposition (IID) aims to separate a natural image into its illumination-invariant components (albedo/reflectance) and illumination-variant components (shade/illumination). This is crucial for various computer vision tasks like image editing and semantic segmentation. Traditional methods often rely on supervised learning with ground truth data, which is difficult to obtain. Unsupervised methods have emerged but often struggle with accurately removing shadows. While recent work (IID-LI) has shown impressive results by incorporating LiDAR intensity, its requirement during inference limits its real-world applicability. This paper addresses the challenge of achieving high-quality IID without the need for LiDAR during inference, thereby expanding its practical use cases.

Methods

LIET builds upon the USI³D architecture but introduces several key innovations:

  • Partially-Shared Model: During training, LIET utilizes both an image encoder path and a LiDAR encoder path, allowing it to learn from both modalities. However, during inference, it exclusively uses the image encoder path.
  • Albedo-Alignment (AA) Loss: This novel loss function aligns the gray-scale albedo inferred from the image with the albedo inferred from LiDAR intensity. Since LiDAR intensity is inherently illumination-invariant, this loss transfers this property to the image-encoder path, effectively reducing cast shadows.
  • Image-LiDAR Conversion (ILC) Paths: These paths facilitate mutual style translation between the image and LiDAR intensity. By treating the image and LiDAR intensity as having shared content but different styles, these paths help disentangle content from style, improving the separation accuracy.
  • Within-Domain Reconstruction: Similar to USI³D, LIET reconstructs inputs within their respective domains (image, LiDAR, albedo, shade) to extract features.
  • Cross-Domain Translation: This component infers albedo and shade from the input image (or LiDAR intensity during training).

Datasets

  • NTT-IID dataset: Used for training. This dataset consists of 10,000 pairs of images and corresponding LiDAR intensities from outdoor scenes, with a subset annotated for IID evaluation.
  • FSVG dataset: Used as the target domain for albedo and shade evaluation.

Key Insights and Ideas

  • LiDAR as a "shadow-free" prior: The core insight is that LiDAR intensity, being unaffected by illumination, acts as a powerful prior for learning to remove shadows from image-based IID.
  • Training vs. Inference separation: The ability to use LiDAR during training but not inference is a critical design choice that balances performance gains with practical usability.
  • Albedo alignment for shadow removal: Directly aligning albedos from image and LiDAR, even in grayscale, is surprisingly effective at transferring illumination-invariance.
  • Content-style disentanglement via cross-modal translation: Leveraging the shared content between image and LiDAR to improve style separation is a clever way to enhance decomposition quality.

Key Results

  • Comparable IID Quality: LIET achieves IID quality comparable to IID-LI (which requires LiDAR during inference) when using only a single image during inference.
  • Reduced Shadows: Qualitative results show significantly fewer noticeable shadows in the inferred albedos from LIET compared to models that do not utilize LiDAR intensity.
  • Improved Image Quality: LIET consistently achieved the highest scores across five different Image Quality Assessment (IQA) metrics, indicating better visual quality of the inferred albedos compared to other top models.
  • Ablation Study: The study confirmed the effectiveness of the AA loss and ILC paths, showing improvements in IID quality when these components are included.

Limitations and Future Work

  • Lambertian Surfaces: The current work focuses on Lambertian surfaces. The authors suggest that future work could extend LIET's applicability to diffuse-specular mixed surfaces by incorporating additional domain data like hyperspectral images or multi-bounce LiDAR.
  • Bias in Annotation: The paper notes that models inferring flatter albedos have an advantage due to annotation biases in certain datasets. While LIET performs well, it's acknowledged that methods assuming local albedo flatness might show slightly better results on specific metrics due to this bias.

Summary updated

BibTeX
@inproceedings{sato2025unsupervised,
  author = {Shogo Sato and Takuhiro Kaneko and Kazuhiko Murasaki and Taiga Yoshida and Ryuichi Tanida and Akisato Kimura},
  title = {{Unsupervised Single-Image Intrinsic Image Decomposition with LiDAR Intensity Enhanced Training}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2025},
}
Figure