FCNet: A Convolutional Neural Network for Arbitrary-Length Exposure Estimation

Liang Jin, Anran Zhang, Jun Xu, Hui Li, Xiantong Zhen, Zhen, Xiantong

TL;DR

This paper introduces FCNet, a novel convolutional neural network that can simultaneously handle single-exposure correction (SEC) and multi-exposure fusion (MEF) for image sequences of arbitrary lengths. By leveraging a Laplacian pyramid decomposition and a unified fusion-correction framework, FCNet effectively addresses the limitations of existing methods that are specialized for either SEC or MEF, particularly in challenging scenarios with only under- or over-exposed images.

What Problem Does This Paper Address?

Digital photographs often suffer from over- or under-exposure, leading to a loss of visual quality. Existing solutions fall into two main categories: Single-Exposure Correction (SEC) aims to fix a single poorly exposed image, while Multi-Exposure Fusion (MEF) combines multiple images of the same scene with different exposures to create a well-exposed image. However, these approaches are typically developed independently, ignoring the inherent correlation between them. This leads to several issues:

  • Specialization: SEC methods can't fuse multiple images, and MEF methods often struggle with sequences containing only under- or over-exposed images, as they lack explicit correction capabilities.
  • Arbitrary Length Sequences: Existing methods are often designed for a fixed number of input images, making them inflexible for sequences of varying lengths.
  • Challenging Scenarios: MEF methods can fail when all images in a sequence are either under- or over-exposed, as there's no complementary information to fuse.

FCNet aims to bridge this gap by providing a unified framework that can handle both SEC and MEF tasks for image sequences of any length, including single images.

Methods

FCNet employs a novel Fusion-Correction Network (FCNet) built upon a Laplacian Pyramid (LP) decomposition framework. This approach allows the network to process images at multiple scales, from coarse to fine.

The core of FCNet consists of Fusion-Correction (FC) blocks. Each FC block operates on the low-frequency base component of an image sequence at a specific LP level. It performs two main operations sequentially:

  1. Fusion Block: This lightweight block predicts weight maps for the input image sequence. These weights are then used to fuse the sequence into a single image. This fusion is designed to be efficient, using a "Downsample-Execute-Upsample" scheme to reduce computational costs, especially for long sequences.
  2. Correction Block: This block, designed as a U-Net-like network, takes the fused image and enhances its exposure. The U-Net architecture with skip connections is chosen for its effectiveness in low-level vision tasks.

The output of an FC block at a given LP level is then upsampled and combined with the high-frequency details from the next finer LP level. This process is repeated across all LP levels, allowing for coarse-to-fine processing.

FCNet is trained end-to-end using a loss function that combines:

  • Pyramid Reconstruction Loss (Lpr): This loss supervises the output at each LP level by comparing it to the corresponding layer of a ground-truth image's Gaussian pyramid.
  • Pyramid Spatial Consistency Loss (Lps): This loss encourages consistency between neighboring regions in the output and ground-truth images, improving robustness on homogeneous regions.

Datasets

The paper utilizes a dataset from [1], which contains 24,330 images. This dataset is described as the largest available with multi-exposed image sequences. The images are rendered with 5 digital Exposure Values (EVs: -1.5, -1, +0, +1, +1.5). The dataset is divided into:

  • Training set: 17,675 images
  • Validation set: 750 images
  • Test set: 5,905 images

The ground-truths used are images retouched by "Expert C" from the MIT-Adobe FiveK dataset [46].

Key Insights and Ideas

  • Unified Framework: The most significant insight is the successful integration of Single-Exposure Correction (SEC) and Multi-Exposure Fusion (MEF) into a single, coherent network. This addresses the limitations of specialized methods.
  • Laplacian Pyramid for Multi-Scale Processing: Using the Laplacian pyramid decomposition allows FCNet to effectively capture and process image information at different scales, which is crucial for both fusion and correction tasks.
  • Fusion-Correction Synergy: The sequential application of fusion and correction within each LP level, and the overall LP framework, creates a synergistic effect. Fusion provides a more robust base for correction, and correction refines the fused output.
  • Arbitrary Length Handling: The network's ability to process sequences of any length, including single images, is a key practical advantage. This is achieved by concatenating images along the batch dimension and adapting the fusion module.
  • Loss Function Design: The combination of pyramid reconstruction and spatial consistency losses effectively guides the network to produce well-exposed and visually pleasing results, mitigating artifacts and improving robustness.

Key Results

  • Performance: FCNet achieves comparable or better performance than state-of-the-art methods on both MEF and SEC tasks.
  • On MEF tasks (Under-EF, Over-EF, All MEF), FCNet consistently achieves competitive or superior PSNR and SSIM scores compared to methods like Mertens09, MEF-GAN, and MEF-Net.
  • On SEC tasks, FCNet achieves comparable PSNR/SSIM results with MSEC [1] and better results than other SEC methods like LIME, WVM, EnlightenGAN, and Zero-DCE.
  • Qualitative Results: FCNet produces visually pleasing results that are closer to the ground-truth, preserving color and contrast information. It effectively handles challenging scenarios where other methods fail, such as sequences with only under- or over-exposed images.
  • Efficiency: FCNet is generally faster than many comparison methods, especially for higher resolutions, due to its LP decomposition and efficient fusion block design.

Limitations and Future Work

  • Computational Cost: While FCNet is efficient, processing high-resolution images, especially long sequences, can still be computationally intensive. The paper notes that changing the order of fusion and correction blocks in all LP levels was prohibitive due to memory constraints.
  • Model Size vs. Performance: The ablation studies show that while larger FC blocks generally lead to better performance, there's a trade-off with model size and computational cost. The "Large→Small" design (larger blocks at lower LP levels, smaller at higher) offers a good balance.
  • Future Work: The authors suggest further exploration of the choice of $\lambda$ for balancing the loss terms and investigating different network design schemes. The paper also implicitly points to the potential for further optimization of the fusion and correction blocks for even greater efficiency and performance.

Summary updated

Abstract

The photographs captured by digital cameras usually suffer from over or under exposure problems. For image exposure enhancement, the tasks of Single-Exposure Correction (SEC) and Multi-Exposure Fusion (MEF) are widely studied in the image processing community. However, current SEC or MEF methods are developed under different motivations and thus ignore the internal correlation between SEC and MEF, making it difficult to process arbitrary-length sequences with improper exposures. Besides, the MEF methods usually fail at estimating the exposure of a sequence containing only under-exposed or over-exposed images. To alleviate these problems, in this paper, we develop a novel Fusion-Correction Network (FCNet) to tackle an arbitrary-length (including one) image sequence with improper exposures. This is achieved by fusing and correcting an image sequence by Laplacian Pyramid (LP) image decomposition. In each LP level, the low-frequency base component of the input image sequence is fed into a Fusion block and a Correction block sequentially for consecutive exposure estimation, implemented by alternative exposure fusion and correction. The exposure-corrected image in current LP level is upsampled and fused with the high-frequency detail components of the input image sequence in the next LP level, to output the base component for the Fusion and Correction blocks in next LP level. Experiments on the benchmark dataset demonstrate that our FCNet is effective on arbitrary-length exposure estimation, including both SEC and MEF. The code is publicly released at https://github.com/NKUJinLiang/FCNet.
BibTeX
@misc{jin2022fcnet,
  author = {Liang Jin and Anran Zhang and Jun Xu and Hui Li and Xiantong Zhen and Zhen, Xiantong},
  title = {{FCNet: A Convolutional Neural Network for Arbitrary-Length Exposure Estimation}},
  year = {2022},
  eprint = {2203.03624},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2203.03624},
}
Figure