TL;DR
This paper introduces FCNet, a novel convolutional neural network that can simultaneously handle single-exposure correction (SEC) and multi-exposure fusion (MEF) for image sequences of arbitrary lengths. By leveraging a Laplacian pyramid decomposition and a unified fusion-correction framework, FCNet effectively addresses the limitations of existing methods that are specialized for either SEC or MEF, particularly in challenging scenarios with only under- or over-exposed images.
What Problem Does This Paper Address?
Digital photographs often suffer from over- or under-exposure, leading to a loss of visual quality. Existing solutions fall into two main categories: Single-Exposure Correction (SEC) aims to fix a single poorly exposed image, while Multi-Exposure Fusion (MEF) combines multiple images of the same scene with different exposures to create a well-exposed image. However, these approaches are typically developed independently, ignoring the inherent correlation between them. This leads to several issues:
- Specialization: SEC methods can't fuse multiple images, and MEF methods often struggle with sequences containing only under- or over-exposed images, as they lack explicit correction capabilities.
- Arbitrary Length Sequences: Existing methods are often designed for a fixed number of input images, making them inflexible for sequences of varying lengths.
- Challenging Scenarios: MEF methods can fail when all images in a sequence are either under- or over-exposed, as there's no complementary information to fuse.
FCNet aims to bridge this gap by providing a unified framework that can handle both SEC and MEF tasks for image sequences of any length, including single images.
Methods
FCNet employs a novel Fusion-Correction Network (FCNet) built upon a Laplacian Pyramid (LP) decomposition framework. This approach allows the network to process images at multiple scales, from coarse to fine.
The core of FCNet consists of Fusion-Correction (FC) blocks. Each FC block operates on the low-frequency base component of an image sequence at a specific LP level. It performs two main operations sequentially:
- Fusion Block: This lightweight block predicts weight maps for the input image sequence. These weights are then used to fuse the sequence into a single image. This fusion is designed to be efficient, using a "Downsample-Execute-Upsample" scheme to reduce computational costs, especially for long sequences.
- Correction Block: This block, designed as a U-Net-like network, takes the fused image and enhances its exposure. The U-Net architecture with skip connections is chosen for its effectiveness in low-level vision tasks.
The output of an FC block at a given LP level is then upsampled and combined with the high-frequency details from the next finer LP level. This process is repeated across all LP levels, allowing for coarse-to-fine processing.
FCNet is trained end-to-end using a loss function that combines:
- Pyramid Reconstruction Loss (Lpr): This loss supervises the output at each LP level by comparing it to the corresponding layer of a ground-truth image's Gaussian pyramid.
- Pyramid Spatial Consistency Loss (Lps): This loss encourages consistency between neighboring regions in the output and ground-truth images, improving robustness on homogeneous regions.
Datasets
The paper utilizes a dataset from [1], which contains 24,330 images. This dataset is described as the largest available with multi-exposed image sequences. The images are rendered with 5 digital Exposure Values (EVs: -1.5, -1, +0, +1, +1.5). The dataset is divided into:
- Training set: 17,675 images
- Validation set: 750 images
- Test set: 5,905 images
The ground-truths used are images retouched by "Expert C" from the MIT-Adobe FiveK dataset [46].
Key Insights and Ideas
- Unified Framework: The most significant insight is the successful integration of Single-Exposure Correction (SEC) and Multi-Exposure Fusion (MEF) into a single, coherent network. This addresses the limitations of specialized methods.
- Laplacian Pyramid for Multi-Scale Processing: Using the Laplacian pyramid decomposition allows FCNet to effectively capture and process image information at different scales, which is crucial for both fusion and correction tasks.
- Fusion-Correction Synergy: The sequential application of fusion and correction within each LP level, and the overall LP framework, creates a synergistic effect. Fusion provides a more robust base for correction, and correction refines the fused output.
- Arbitrary Length Handling: The network's ability to process sequences of any length, including single images, is a key practical advantage. This is achieved by concatenating images along the batch dimension and adapting the fusion module.
- Loss Function Design: The combination of pyramid reconstruction and spatial consistency losses effectively guides the network to produce well-exposed and visually pleasing results, mitigating artifacts and improving robustness.
Key Results
- Performance: FCNet achieves comparable or better performance than state-of-the-art methods on both MEF and SEC tasks.
- On MEF tasks (Under-EF, Over-EF, All MEF), FCNet consistently achieves competitive or superior PSNR and SSIM scores compared to methods like Mertens09, MEF-GAN, and MEF-Net.
- On SEC tasks, FCNet achieves comparable PSNR/SSIM results with MSEC [1] and better results than other SEC methods like LIME, WVM, EnlightenGAN, and Zero-DCE.
- Qualitative Results: FCNet produces visually pleasing results that are closer to the ground-truth, preserving color and contrast information. It effectively handles challenging scenarios where other methods fail, such as sequences with only under- or over-exposed images.
- Efficiency: FCNet is generally faster than many comparison methods, especially for higher resolutions, due to its LP decomposition and efficient fusion block design.
Limitations and Future Work
- Computational Cost: While FCNet is efficient, processing high-resolution images, especially long sequences, can still be computationally intensive. The paper notes that changing the order of fusion and correction blocks in all LP levels was prohibitive due to memory constraints.
- Model Size vs. Performance: The ablation studies show that while larger FC blocks generally lead to better performance, there's a trade-off with model size and computational cost. The "Large→Small" design (larger blocks at lower LP levels, smaller at higher) offers a good balance.
- Future Work: The authors suggest further exploration of the choice of $\lambda$ for balancing the loss terms and investigating different network design schemes. The paper also implicitly points to the potential for further optimization of the fusion and correction blocks for even greater efficiency and performance.
Summary updated