Modeling the Lighting in Scenes As Style for Auto White-Balance Correction

Furkan Kınlı, Doğa Yılmaz, Barış Özcan, Furkan KıraçWACV 2023

TL;DR

This paper proposes a novel approach to automatic white-balance (AWB) correction by reframing the problem as modeling scene lighting as a "style" factor. By treating lighting variations as style, their method learns to generate weighting maps that effectively blend different white-balanced versions of an image, achieving improved results, especially in challenging mixed-illuminant scenarios, without explicit illuminant estimation.

What Problem Does This Paper Address?

Accurate color reproduction in images is crucial, and automatic white-balance (AWB) is a fundamental image processing step that aims to correct for the color cast introduced by different light sources. While AWB works well under single, uniform lighting conditions, it struggles significantly in scenes with multiple, distinct light sources (mixed illuminants). Traditional AWB methods often rely on estimating a single global illuminant color, which fails to capture the complexity of mixed lighting. This leads to color casts and inaccurate color reproduction in various parts of the image. Existing methods for mixed-illuminant AWB often involve complex illuminant estimation or rely on blending pre-defined WB settings, but they still face challenges in achieving perceptually pleasing results. This paper tackles the problem of improving AWB, particularly for mixed-illuminant scenes, by proposing a new perspective that leverages the concept of "style" from computer vision.

Methods

The core of the proposed method is a novel AWB strategy that models scene lighting as a "style" factor. Instead of directly estimating illuminant colors, the network learns to generate pixel-wise weighting maps for different white-balance (WB) settings.

  1. Style Extractor Module: The authors adapt a style extraction module, inspired by style transfer literature, to process multiple versions of the same scene rendered with different WB settings. This module, based on a multi-head mapping network and a VGG encoder, learns to extract "style" information related to the lighting conditions.
  2. Adaptive Normalization: This extracted style information is then used to adaptively normalize feature maps within an encoder-decoder network. This normalization step, using Adaptive Instance Normalization (AdaIN), effectively disentangles the "style" (lighting) from the "content" of the image.
  3. Weighting Map Generation: The network, after processing the style-informed features, learns to generate weighting maps for each WB setting. These maps indicate how much each WB-corrected version of the image should contribute to the final output.
  4. Blending: The final AWB corrected image is obtained by linearly combining the small, pre-rendered images (each with a specific WB setting) using the learned weighting maps. This approach avoids explicit illuminant estimation.
  5. Post-processing: Two post-processing steps are applied to further refine the weighting maps: multi-scale ensembling and edge-aware smoothing (EAS).

Datasets

  • Training: RenderedWB dataset [5] (65,000 sRGB images with specific WB settings).
  • Evaluation:
    • Cube+ dataset [9] (single-illuminant).
  • MIT-Adobe FiveK dataset [18, 19] (single-illuminant).
  • Mixed-illuminant evaluation set proposed by [4] (150 synthetic images).
  • Night photography rendering set [26].

Key Insights and Ideas

  • Lighting as Style: The central idea that scene lighting, especially in mixed-illuminant scenarios, can be effectively modeled as a "style" factor is a significant conceptual shift. This reframing allows leveraging powerful style transfer techniques for AWB.
  • Disentangling Lighting: By using style extraction and adaptive normalization, the method implicitly disentangles the lighting "style" from the image content, enabling more precise control over color correction.
  • No Explicit Illuminant Estimation: The proposed method bypasses the often challenging and error-prone step of estimating illuminant colors, particularly for mixed lighting, which is a major advantage.
  • Interpretable Weighting Maps: The generated weighting maps are shown to be more detailed and interpretable than those from prior methods, accurately reflecting the spatial distribution of different illuminants.

Key Results

  • Improved Performance: The proposed method achieves competitive and often superior results compared to recent AWB methods on both single- and mixed-illuminant datasets, as measured by MSE, MAE, and ΔE 2000 metrics.
  • Mixed-Illuminant Advantage: The method shows particular strength in handling mixed-illuminant scenes, producing more natural and visually pleasing results (demonstrated in Figure 3 and 5).
  • Ablation Study: The ablation study (Table 3) confirms the benefit of multi-scale weighting maps and edge-aware smoothing for improving performance.
  • Qualitative Improvements: Visual comparisons (Figure 2, 3, 4, 5) highlight the method's ability to produce more accurate colors and avoid color casts, especially in complex lighting scenarios.

Limitations and Future Work

  • Memory Overhead: The authors acknowledge that their style extraction module has a larger memory overhead compared to some recent methods.
  • Computational Complexity: While not explicitly stated as a limitation, the use of VGG and multiple AdaIN layers suggests a potentially higher computational cost during training.
  • Future Work: The authors suggest designing a style extraction module with lower memory overhead without sacrificing performance. They also note that incorporating total variation regularization into the loss function could potentially achieve similar smoothing effects as post-processing, simplifying the pipeline.

Summary updated

BibTeX
@inproceedings{knl2023modeling,
  author = {Furkan Kınlı and Doğa Yılmaz and Barış Özcan and Furkan Kıraç},
  title = {{Modeling the Lighting in Scenes As Style for Auto White-Balance Correction}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2023},
}
Figure