DARN: A Deep Adversarial Residual Network for Intrinsic Image Decomposition

Louis Lettry, Kenneth Vanhoey, Luc Van GoolWACV 2018 · DOI

TL;DR

This paper introduces DARN, a novel deep learning architecture for intrinsic image decomposition that jointly estimates albedo and shading. By leveraging a residual GAN and a perceptually motivated loss, DARN achieves state-of-the-art results, producing more accurate and consistent decompositions than previous methods, crucial for applications like relighting and 3D reconstruction.

What Problem Does This Paper Address?

Understanding how an image is formed is a fundamental problem in computer vision. An image is essentially a product of the scene's inherent properties (like surface color, or "albedo") and how light interacts with it (the "shading"). Reversing this process, known as intrinsic image decomposition, is vital for many tasks. For instance, separating albedo allows for relighting scenes in augmented reality, while shading information can aid in shape estimation.

Traditional methods relied on strong physical assumptions (like smooth shading or sparse albedo) or required external data like depth maps. While deep learning has shown promise, existing approaches often struggle with:

  • Inconsistent predictions: The predicted albedo and shading don't perfectly multiply back to the original image (I ≠ A * S).
  • Scale ambiguity: The absolute scale of albedo and shading is hard to determine from a single image, requiring manual post-processing.
  • Artifacts: Common issues include blurriness, color bleeding, and contrast oscillations.
  • Reliance on external data: Some methods require depth information, limiting their applicability.

This paper aims to address these limitations by proposing a fully data-driven, end-to-end deep learning solution that produces consistent, scale-aware intrinsic decompositions.

Methods

The core of DARN is a deep generative adversarial network (GAN) designed for joint albedo and shading estimation. Here's a breakdown of their key methodological contributions:

  • Fully Joint Learning: Instead of predicting albedo and shading independently, DARN learns them simultaneously. This interdependence helps enforce consistency.
  • Residual Learning: Inspired by ResNet, the network learns the deviation from the input image rather than a direct mapping. This is beneficial because the input image shares significant information with both albedo and shading.
  • Adversarial Loss: To combat typical GAN artifacts like blurriness and color bleeding, DARN employs two discriminator networks (one for albedo, one for shading). These discriminators act as a perceptual loss, pushing the generator to produce outputs that are indistinguishable from ground truth to a trained classifier.
  • Consistent Decomposition by Construction: The architecture is designed such that the predicted albedo and shading always multiply back to the original image (I = A * S). This is achieved by predicting one component (e.g., shading) and then deriving the other (albedo) through element-wise division of the input image by the predicted shading. This ensures energy preservation.
  • New Scale-Aware Metrics: Recognizing that absolute albedo has physical meaning and that scale consistency is important, the authors introduce new metrics (like rs-MSE) that evaluate the decomposition's accuracy while accounting for relative scale.

Datasets

  • MPI Sintel Dataset: This dataset, derived from a synthetic animation movie, is used for training and evaluation. It provides ground truth albedo and shading layers. The authors note that the original Sintel data doesn't perfectly adhere to I = A * S, so they "recompose" it to create a consistent dataset called "ResynthSintel" for training and testing.
  • MIT Intrinsic Images Dataset: This dataset contains realistic intrinsic images of single objects but is small and lacks variety, making it unsuitable for training alone. It's used in conjunction with Sintel for evaluating inter-domain generalization.
  • Intrinsic Images in the Wild (IIW): A large-scale real-world dataset with sparse ground truth annotations. It's used to highlight the limitations of the model's generalization to real-world data.

Key Insights and Ideas

  • Joint learning is crucial for consistency: Treating albedo and shading as independent problems leads to inconsistencies. Learning them together, with an architecture that enforces I = A * S by construction, is a significant step towards more usable decompositions.
  • Adversarial training as a perceptual loss: Using discriminators to guide the generator is highly effective at mitigating common CNN artifacts, leading to sharper and cleaner results.
  • Absolute albedo matters: The authors argue that albedo has a physical meaning and should be treated as an absolute value, not just a relative one. This motivates their focus on energy-preserving decomposition.
  • Scale-aware metrics are necessary: Traditional scale-invariant metrics can be misleading. New metrics that account for the inherent scale ambiguity and the importance of I = A * S are needed for a more thorough evaluation.

Key Results

  • State-of-the-art performance: DARN outperforms previous deep learning methods on the Sintel dataset across various metrics, including scale-invariant (si-MSE, si-LMSE) and perceptual (DSSIM) measures.
  • Improved sharpness and consistency: Qualitative results show DARN produces significantly sharper albedo and more consistent shading compared to prior work, with fewer artifacts.
  • Better generalization (within domains): The model demonstrates improved intra-domain generalization, particularly on the more challenging "scene split" evaluation.
  • Limited inter-domain generalization: While mixing datasets (Sintel + MIT) improves generalization to some extent, the model still struggles with real-world data (IIW) due to its unique biases and lack of dense ground truth.

Limitations and Future Work

  • Real-world data generalization: The primary limitation is the model's difficulty in generalizing to real-world images, which have different characteristics (e.g., white balancing, sensor noise) than the synthetic Sintel dataset. The authors acknowledge that the current GT datasets are either unrealistic, small, or sparse.
  • Future work directions:
    • Developing domain shift techniques or entirely new approaches to tackle dense intrinsic decomposition of real-world data.
  • Investigating the impact of optics and acquisition devices on shading.
  • Further exploring and refining scale-aware metrics.
  • Creating more diverse and realistic dense ground truth datasets.

Summary updated

BibTeX
@inproceedings{lettry2018darn,
  author = {Louis Lettry and Kenneth Vanhoey and Luc Van Gool},
  title = {{DARN: A Deep Adversarial Residual Network for Intrinsic Image Decomposition}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2018},
  doi = {10.1109/wacv.2018.00153},
}
Figure