Abstract
Multi-exposure image fusion (MEF) is essential for capturing high-dynamic-range (HDR) scenes using standard digital sensors. However, conventional spatial-domain deep learning models often struggle with global illumination consistency and halo artifacts due to their limited receptive fields. This paper introduces MEF-SFI, a framework that integrates spatial and frequency domain features. By utilizing the Discrete Fourier Transform (DFT), the network achieves a global receptive field to better model illumination. Our results show superior performance across multiple metrics and visual benchmarks compared to state-of-the-art methods.
1. Introduction
Digital imaging sensors frequently fail to capture the full dynamic range of natural scenes, resulting in over-exposed highlights or under-exposed shadows. Multi-exposure image fusion (MEF) addresses this by merging a sequence of low-dynamic-range (LDR) images taken at different exposure levels. While deep learning has significantly improved MEF, most architectures rely exclusively on spatial convolutions. These local operations lack the global context necessary to ensure exposure consistency across the entire frame, often leading to unsightly halo artifacts and inconsistent lighting.
2. Methodology
The MEF-SFI framework employs a dual-path architecture to balance local detail and global coherence.
2.1 Spatial and Frequency Paths
- Spatial Path: Utilizes a DenseNet-based architecture to extract fine-grained semantic features and local textures from the input images.
- Frequency Path: Employs the Discrete Fourier Transform (DFT) to convert features into the frequency domain. Unlike Transformers, which achieve global modeling through computationally expensive self-attention, the frequency path provides an image-wide receptive field with significantly lower computational overhead.
2.2 Spatial-Frequency Fusion Module (SFFM)
The core of the network is the SFFM, which employs spatial and channel attention mechanisms to intelligently weigh and integrate features from both domains. This ensures that the final fusion preserves sharp local edges while maintaining global lighting harmony.
2.3 Dual-Domain Loss
Training is guided by a composite loss function that constrains both spatial pixel differences (via MSSIM and L2-norm) and frequency-domain components (amplitude and phase), ensuring the network learns both structural and illumination properties accurately.
3. Theoretical Motivation: Amplitude and Phase
The integration of the frequency domain is motivated by the distinct roles of amplitude and phase in the Fourier transform. In image processing, the amplitude component primarily carries luminance (brightness) information, while the phase component encodes structural details.
As demonstrated in Figure 1, swapping the amplitude of an over-exposed image with the phase of an under-exposed image creates a "pseudo-image" that retains the structure of the latter but adopts the brightness of the former. MEF-SFI leverages this property to treat the frequency domain as a specialized channel for balancing global exposure.
4. Experiments and Results
The model was trained and evaluated on the PQA-MEF dataset, a pre-aligned version of the SICE dataset scaled to 1/4 resolution.
4.1 Quantitative Evaluation
MEF-SFI was benchmarked against eight state-of-the-art methods, including DeepFuse, U2Fusion, and HoLoCo. It achieved superior scores in key metrics:
- MEF-SSIM: 0.969
- MSSIM: 0.954
- Yang’s Metric ($Q_Y$): 0.887
4.2 Qualitative Analysis
Visually, MEF-SFI produces more natural results than its competitors. It effectively eliminates "halo" artifacts around high-contrast boundaries and maintains superior color saturation in both extreme highlights (e.g., bright skies) and deep shadows (e.g., mountain textures).
5. Conclusion and Future Work
MEF-SFI demonstrates that integrating frequency-domain information significantly enhances the quality of multi-exposure fusion by providing a global context that spatial-only models lack. While the current model primarily focuses on the luminance (Y) channel of the YCbCr color space, future work will explore more sophisticated integration of chrominance (Cb, Cr) channels and further optimization for mobile deployment.
Summary updated