Learning to Correct Overexposed and Underexposed Photos

Mahmoud Afifi, Konstantinos G. Derpanis, Björn Ommer, Michael S. BrownarXiv 2020 · OpenReview

TL;DR

This paper introduces a novel deep learning model that can correct both overexposed and underexposed photos using a single, unified approach. They also present a large, diverse dataset to train and evaluate their model, which significantly outperforms existing methods, especially for overexposed images.

What Problem Does This Paper Address?

Capturing the perfect exposure is a common challenge in photography. When a photo is either too bright (overexposed) or too dark (underexposed), crucial details and visual appeal are lost. While many existing methods focus on fixing underexposed images, correcting overexposed photos, where details are washed out, has been less effectively addressed. This paper aims to bridge this gap by developing a method that can handle both types of exposure errors comprehensively.

Methods

The authors propose a "coarse-to-fine" deep neural network (DNN) architecture. This approach breaks down the complex problem of exposure correction into two main stages:

  1. Global Color Correction: The network first addresses the overall color balance and brightness of the image, focusing on low-frequency information.
  2. Detail Enhancement: Subsequently, it refines the image by enhancing details in mid- and high-frequency layers.

This is achieved by processing the image through a Laplacian pyramid, which decomposes it into different frequency bands. Each level of the pyramid is processed by a dedicated sub-network (a U-Net-like architecture). The network is trained end-to-end, meaning all parts are optimized simultaneously. They also incorporate an adversarial loss to improve the perceptual realism of the corrected images.

Datasets

The paper introduces a new, large-scale dataset for exposure correction, comprising over 24,000 images. This dataset was generated by rendering raw-RGB images from the MIT-Adobe FiveK dataset with various exposure values (EVs), ranging from -1.5 to +1.5. Each image in this dataset has a corresponding "properly exposed" reference image. This new dataset is notable for its broad range of exposure errors, significantly exceeding the coverage of existing datasets like the low-light (LOL) dataset.

Key Insights and Ideas

  • Unified Over/Under Exposure Correction: The most significant insight is that a single model can effectively handle both over- and underexposure errors, a task that has historically been treated separately.
  • Coarse-to-Fine Strategy: Decomposing the problem into global color correction followed by detail enhancement, using a Laplacian pyramid, proves to be a robust strategy for achieving high-quality results.
  • Importance of Diverse Data: The creation of a large dataset with a wide range of exposure values was crucial for training a model that generalizes well and performs effectively across different types of exposure errors.
  • Adversarial Loss for Realism: The inclusion of an adversarial loss term demonstrably improves the perceptual quality and realism of the corrected images, making them more visually appealing.

Key Results

  • State-of-the-Art Performance: The proposed method achieves results on par with existing state-of-the-art methods for underexposed images and shows significant improvements for overexposed images.
  • Quantitative Metrics: The paper reports strong performance on standard metrics like PSNR, SSIM, and PI, particularly outperforming other methods on overexposed images. For instance, their method achieved the best results across all metrics for overexposed images in their test set.
  • Qualitative Superiority: Visual comparisons (Figs. 1, 7, S5-S9) show that their method produces more natural and detailed results, especially for challenging overexposed images where other methods struggle.
  • Generalization: The model demonstrates good generalization capabilities, performing well on images not seen during training, including those from external datasets and real-world Flickr images.

Limitations and Future Work

  • Lack of Semantic Information: The authors acknowledge that their method struggles in regions with insufficient semantic information (e.g., completely saturated faces), leading to color bleeding. They suggest an interactive control mechanism (scaling pyramid levels) to mitigate this.
  • Noise in Dark Regions: The model can introduce noise when dealing with extreme dark regions in underexposed images.
  • Future Work: The authors suggest exploring the impact of their exposure correction method on various computer vision tasks, such as face detection and semantic segmentation, as a pre-processing step. They also mention the potential for interactive control of the output through scale vectors.

Summary updated

Abstract

Capturing photographs with wrong exposures remains a major source of errors in camera-based imaging. Exposure problems are categorized as either: (i) overexposed, where the camera exposure was too long, resulting in bright and washed-out image regions, or (ii) underexposed, where the exposure was too short, resulting in dark regions. Both under- and overexposure greatly reduce the contrast and visual appeal of an image. Prior work mainly focuses on underexposed images or general image enhancement. In contrast, our proposed method targets both over- and underexposure errors in photographs. We formulate the exposure correction problem as two main sub-problems: (i) color enhancement and (ii) detail enhancement. Accordingly, we propose a coarse-to-fine deep neural network (DNN) model, trainable in an end-to-end manner, that addresses each sub-problem separately. A key aspect of our solution is a new dataset of over 24,000 images exhibiting the broadest range of exposure values to date with a corresponding properly exposed image. Our method achieves results on par with existing state-of-the-art methods on underexposed images and yields significant improvements for images suffering from overexposure errors.
BibTeX
@misc{afifi2020learning,
  author = {Mahmoud Afifi and Konstantinos G. Derpanis and Björn Ommer and Michael S. Brown},
  title = {{Learning to Correct Overexposed and Underexposed Photos}},
  year = {2020},
  eprint = {2003.11596},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2003.11596},
}
Figure