TL;DR
This paper introduces a novel multi-exposure image fusion (MEF) method that significantly improves ghost removal in dynamic scenes by leveraging a priori exposure quality and low-level image features. By intelligently selecting a reference image and decomposing images into base and detail layers, the method achieves more robust and visually appealing high-dynamic-range images, outperforming existing techniques in both static and dynamic scenarios.
What Problem Does This Paper Address?
Capturing the full dynamic range of a scene, especially in challenging lighting conditions, is a persistent problem for imaging devices. Human eyes can perceive a much wider range of brightness and detail than a single camera exposure can capture. Multi-exposure image fusion (MEF) aims to overcome this by combining multiple images of the same scene taken at different exposure levels to create a single high-dynamic-range (HDR) image.
While MEF methods have advanced significantly, two major challenges remain:
- Ghosting in Dynamic Scenes: When there's motion (either of the camera or objects within the scene) between exposures, the fused image can suffer from ghosting artifacts. Existing methods often struggle to effectively remove these ghosts, especially when dealing with complex motion or varying exposure qualities.
- Fusion Quality in Static Scenes: Even in static scenes, achieving optimal fusion that preserves fine details, maintains color saturation, and avoids over/underexposure can be difficult. Many methods focus on either global or local exposure optimization, leading to compromises in overall visual quality.
This paper specifically targets the problem of robust ghost removal in dynamic scenes while also enhancing the fusion quality in static scenes.
Methods
The proposed method, termed Feature Patch-based MEF (FPM), introduces several key innovations:
- A Priori Exposure Quality for Reference Image Selection: Instead of randomly selecting a reference image, FPM first evaluates the exposure quality of each input image. It uses thresholds ($\alpha$ and $\beta$) to identify over/underexposed pixels and selects the image with the best overall exposure quality as the reference. This is crucial for minimizing local exposure distortion and improving ghost removal robustness.
- Image Decomposition using Guided Filter: Each input image is decomposed into a smooth "base layer" (representing large-scale intensity variations) and a "detail layer" (representing small-scale details) using a guided filter. This is a standard technique, but its application here is integrated with other novel steps.
- Image Patch Structure Decomposition: The base layer is further decomposed into three components: signal strength, signal structure, and mean intensity. This allows for more nuanced processing of image patches, considering their structural consistency and strength.
- Global and Local Exposure Optimization: The method optimizes exposure quality at both the global level (using a two-dimensional Gaussian model for the base layer) and the local level (optimizing exposure for image blocks in the detail layer). This dual optimization aims to improve overall image quality and detail preservation.
- Ghost Removal via Structure Consistency and IMF: Ghosting is addressed by checking the structure consistency between patches of the reference image and other source images. The Intensity Map Replacement (IMF) technique is also employed to detect and correct inconsistent pixel points, effectively mitigating ghost artifacts.
Datasets
The paper evaluated its method on a total of 39 sets of multi-exposure source image sequences:
- 24 sets of static scenes.
- 15 sets of dynamic scenes.
The source image sequences were collected from existing datasets used in previous MEF research (Ma [31], Hu [14], and Sen [15]). The paper does not introduce new datasets.
Key Insights and Ideas
- The Importance of a "Good" Reference Image: The paper strongly emphasizes that the quality of the reference image is paramount for effective ghost removal in dynamic scenes. A poorly chosen reference can introduce more artifacts than it removes. Their a priori exposure quality selection directly addresses this.
- Leveraging Low-Level Features for Robustness: By focusing on low-level features like image brightness, contrast, and structure, the FPM method achieves a more fundamental understanding of image content, leading to better fusion and ghost removal compared to methods relying solely on higher-level or more complex features.
- Synergy of Decomposition and Optimization: The combination of spatial-domain decomposition (base/detail layers) with patch-level structure decomposition and both global/local exposure optimization creates a powerful framework that addresses multiple aspects of image fusion simultaneously.
- Ghost Removal as a Precursor to Fusion: The method integrates image registration and ghost removal as a critical preprocessing step before the actual fusion, ensuring that the subsequent fusion process operates on cleaner, more consistent data.
Key Results
The proposed FPM method demonstrated superior performance compared to six other state-of-the-art MEF methods across various metrics:
- Ghost Removal: FPM effectively removed ghosts in dynamic scenes, a significant improvement over methods like DSIFT-EF, EFM, Fast-expo, and FMMR, which often retained ghosting. While APS and SPD-MEF also removed ghosts, FPM achieved better overall visual quality.
- Static Scene Performance: In static scenes, FPM excelled in contrast, local saturation, and detail preservation, often outperforming other methods.
- Objective Evaluation:
- IQA (Image Quality Assessment): FPM achieved the highest average IQA score (0.9746), indicating superior structure consistency with source images.
- QAB/F (Gradient-based Quality Index): FPM obtained the best QAB/F score (0.7411), signifying better preservation of edge information.
- MI (Mutual Information): FPM achieved the highest MI score (1.8551), indicating better similarity between the fused and source images.
- Subjective Evaluation: Human visual system evaluations confirmed that FPM produced fused images with better color saturation, sharpness, and local detail processing, especially in dynamic scenes.
Limitations and Future Work
- Computational Efficiency: The paper acknowledges that the proposed fusion method has relatively low computational efficiency. This is attributed to the extensive calculations involved in a priori exposure quality estimation, structure consistency detection for ghost removal, and global/local exposure optimization. The ghost detection and removal process alone accounted for about 60% of the total computation time.
- Future Work: To address the computational cost, the authors plan to:
- Design a more concise motion-consistency detection algorithm to reduce ghost detection computation.
- Simplify the fusion algorithm parameters and optimize calculation steps for static scene image fusion to improve overall efficiency.
Summary updated