TL;DR
This paper introduces MLP-MFF, a novel, lightweight, and end-to-end deep learning model for multi-focus image fusion. It overcomes the computational burden and input size limitations of existing Transformer/Mamba-based methods by leveraging a Pyramid Fusion MLP architecture, achieving state-of-the-art fusion quality with remarkable efficiency.
What Problem Does This Paper Address?
Modern cameras have a limited depth of field (DoF), meaning only a specific plane in a scene is in focus, while other areas appear blurred. This is a significant limitation for applications requiring clear images across the entire scene, such as microscopy, machine vision, and industrial inspection. Multi-focus image fusion (MFF) aims to solve this by combining multiple images of the same scene, captured at different focal planes, into a single, all-in-focus image.
While deep learning has shown great promise in MFF, current methods face challenges:
- CNNs: Struggle with long-range dependencies.
- Transformers/Mambas: Are computationally expensive and often require patch-based processing, which compromises their ability to capture global context.
- Rigid Input Sizes: Many advanced models can only handle fixed input resolutions, necessitating compromises for higher-resolution images.
This paper addresses the need for an MFF method that is both highly effective and computationally efficient, capable of handling flexible input sizes and capturing global context without excessive computational cost.
Methods
The core of MLP-MFF is built upon the Pyramid Fusion Multi-Layer Perceptron (PFMLP) architecture. This pure MLP-based backbone is designed to:
- Handle Flexible Input Scales: Unlike many other advanced architectures, it doesn't require fixed input sizes.
- Capture Global Receptive Field: It can process the entire image to understand global context.
- Learn Multi-Scale Features: It effectively extracts information at different resolutions.
Key components of MLP-MFF include:
- Pyramid Fusion (PF) Module: Uses multiple pooling branches with learnable pooling layers to create a feature pyramid, capturing local cues at varying scales.
- Single Scale (SS) Module: A variant of channel MLPs that aggregates information, with a specialized down-sampling version that uses depth-wise convolutions to preserve spatial information.
- Dual-Path Adaptive Multi-scale Feature-Fusion Module (DAMFFM-HA): This novel module is crucial for fusing features from the two input images. It employs:
- Hybrid Attention: Combines channel attention (for global feature dependencies) and spatial attention (for highlighting important regions).
- Adaptive Weight Allocation: Dynamically learns weights to fuse features from the two input paths, improving fusion quality.
- End-to-End Architecture: The entire network, from input images to the final fused image, is trained as a single unit, minimizing error accumulation.
- Progressive Reconstruction with Skip Connections: A decoder with skip connections ensures that fine-grained spatial details are preserved during reconstruction.
Datasets
- Training: DUTS dataset (15,572 images) was used for training and validation.
- Evaluation: The model was evaluated on three prominent MFF benchmarks:
- Lytro dataset [28]: 20 light-field camera image pairs.
- MFFW dataset [62]: 13 image pairs with pronounced defocus.
- MFI-WHU dataset [41]: 120 synthetically generated images (via Gaussian blur).
- Road-MF dataset [3]: Used to demonstrate applicability in autonomous driving scenarios.
Key Insights and Ideas
- MLPs as a Viable Alternative: The paper strongly advocates for MLPs as a powerful alternative to CNNs and Transformers for vision tasks, particularly when combined with clever architectural designs like pyramid fusion.
- Global Receptive Field without Patching: MLP-MFF achieves a true global receptive field without resorting to patch-wise processing, a significant advantage over many Transformer-based methods.
- Hybrid Attention for Adaptive Fusion: The DAMFFM-HA module's combination of channel and spatial attention, along with adaptive weighting, is key to its superior fusion performance. It effectively balances the need to understand global context and focus on critical local details.
- Efficiency is Key: The research demonstrates that high-quality fusion doesn't necessarily require massive computational overhead, offering a practical solution for resource-constrained environments.
Key Results
- Superior Fusion Quality: MLP-MFF achieves competitive, and often superior, fusion quality compared to state-of-the-art methods across multiple datasets and evaluation metrics (EI, SF, AVG, Qab/f, SSIM, EN, MI, VIF). It ranked highest in six out of eight metrics on the Lytro dataset.
- Exceptional Efficiency:
- Lowest FLOPs: Achieves the lowest FLOPs (0.52G) among compared methods, an 83.12% reduction compared to the previous best (MFF-GAN).
- Fastest Inference: Achieves the fastest inference time (0.01s), an 83.33% improvement over MFF-GAN.
- Lightweight Model: While not the smallest, its model size (1.23M) is significantly smaller than many Transformer-based methods (e.g., SwinMFF at 41.25M).
- Robustness: Demonstrates strong performance under extreme conditions (synthetically blurred images) and challenging scenarios (high-contrast scenes, small objects).
- Handles Multiple Sources: The method can be extended to fuse more than two images sequentially.
Limitations and Future Work
- Synthetic Defocus: The training data relies on synthetically generated defocus, which may not perfectly capture real-world defocus patterns.
- Dual-Source Focus: The current architecture is optimized for fusing two images. Extending it to handle multi-source or multi-modal scenarios will require significant modifications.
Future work includes exploring more advanced training techniques for real-world data and adapting the architecture for multi-source/multi-modal fusion and image stack fusion.
Summary updated