Generation and Recombination for Multifocus Image Fusion With Free Number of Inputs

Huafeng Li, Dan Wang, Yuxin Huang, Yafei Zhang, Zhengtao Yu

This paper, published in late 2023, introduces GRFusion, a framework designed to solve the "hard pixel" problem in multifocus image fusion while allowing for an arbitrary number of input images.

TL;DR

GRFusion is a novel multifocus image fusion framework that breaks the "two-input" limitation of traditional models by processing any number of source images in parallel. It uses a hybrid strategy that combines pixel selection for clear regions with deep-learning-based image generation for ambiguous "hard pixels," resulting in state-of-the-art visual clarity and fewer artifacts.

What Problem Does This Paper Address?

Multifocus image fusion (MFIF) aims to create a single "all-in-focus" image from several shots of the same scene taken with different focus settings. While this sounds straightforward, existing methods face two major hurdles:

  1. The "Two-Input" Constraint: Most current algorithms assume only two source images. To handle more, they use "alternating fusion" (fusing image A and B, then fusing that result with C). This sequential process often leads to information loss and accumulated errors.
  2. The "Hard Pixel" Dilemma: Pixels located at the boundaries of focused regions or in smooth, textureless areas are notoriously difficult to classify as "in-focus" or "out-of-focus."
    • Recombination methods (which pick the best pixels) often create ugly artifacts at these boundaries.
  3. Reconstruction methods (which generate new pixels) often lose fine details and sharpness compared to the original source.

Methods

The authors propose a "Generation and Recombination" (GR) approach that uses three main components:

  • Hard Pixel Detection (HPD): Instead of assuming two images are perfect complements, GRFusion detects the focus property of each image independently. By comparing these independent maps, the system identifies "hard pixels"—areas where the focus is inconsistent or ambiguous across all inputs.
  • Full-focus Image Generation (FFIG-MFE): For those tricky hard pixels, the model doesn't just guess; it generates a high-quality full-focus version. It uses a Multi-directional Feature Embedding (MFE) module with eight directional edge-extraction kernels to capture fine structural details that standard convolutions might miss.
  • Fusion Result Construction (FRC-G-HP): This is the "recombination" phase. The algorithm acts like a smart editor: if a pixel is clearly in focus in a source image, it takes that original pixel to preserve maximum detail. If a pixel is identified as a "hard pixel," it swaps in the generated version from the FFIG-MFE module to ensure a smooth, artifact-free transition.

Datasets

The model was trained on a synthetic dataset generated from 200 images of the ImageNet (ILSVRC2012) collection, using Gaussian filters to simulate various blur levels.

For evaluation, the authors used three standard benchmarks:

  • Lytro: 20 pairs of multi-focus images and 4 sets of triple-focus images.
  • MFI-WHU: A large-scale dataset containing 120 pairs of images.
  • MFFW: A complex dataset featuring 13 pairs and several multi-input sets (up to 6 images of the same scene).

Key Insights and Ideas

  • Parallelism over Sequentialism: By treating each input image as an independent source of information rather than a pair, the model scales naturally to any number of inputs without needing to change the architecture.
  • The Hybrid Advantage: The "aha moment" here is realizing that you don't have to choose between selecting pixels and generating them. By using generation as a targeted "safety net" specifically for boundary pixels, you get the sharpness of raw data with the smoothness of deep learning.
  • Multi-directional Gradients: Standard CNNs can struggle with edge orientation. By explicitly using eight directional kernels (horizontal, vertical, and various diagonals), the model preserves the geometric integrity of objects at focus boundaries.

Key Results

GRFusion was compared against 9 state-of-the-art methods (including SwinFusion and SESF).

  • Quantitative: It achieved the best or second-best scores across five major metrics ($Q_{MI}, Q_{AB/F}, Q_{CB}, Q_{NCIE}, Q_{SSIM}$) on both the Lytro and MFI-WHU datasets.
  • Qualitative: Visual inspections (and difference maps) showed that GRFusion produced significantly fewer artifacts at the boundaries of objects. In tests with 3, 4, and 6 input images, it maintained high visual quality where alternating fusion methods began to show "chromatic aberrations" or blurring.

Limitations and Future Work

While the paper demonstrates superior performance, a few areas remain for exploration:

  • Computational Complexity: The use of multiple MSFA (Multi-scale Feature Aggregation) modules and directional kernels likely increases the computational footprint compared to simpler models.
  • Real-world Training: The model relies on synthetically blurred images for training; using real-world defocus data (which includes complex lens bokeh) could further improve robustness.
  • Video Fusion: The authors suggest that the "free number of inputs" logic could potentially be extended to temporal fusion in video, though this wasn't tested in the current paper.

Summary updated

Abstract

Multifocus image fusion is an effective method to overcome the limitations of optical lenses. The fused results can be obtained from some existing methods by generating decision maps. However, such methods assume that the focused areas of the two source images are complementary, making it impossible to achieve the simultaneous fusion of multiple images. Additionally, existing methods ignore the impact of hard pixels on the fusion performance, limiting the visual quality improvement of fusion images. To address these issues, a combined generation and recombination model called GRFusion is proposed. In GRFusion, the focus property detection of each source image can be independently implemented, enabling the simultaneous fusion of multiple source images and avoiding information loss caused by alternating fusion. It renders the GRFusion free from the limitation of the number of input images. Furthermore, GRFusion investigates the detection of hard pixels with ambiguous focus properties by analyzing the inconsistencies among the detection results of the focus areas in the source images. This allows the hard pixels to be distinguished from the source images. Besides, a multidirectional gradient embedding method is proposed for generating full-focus images. Subsequently, a hard-pixel-guided recombination mechanism for constructing the fused result is devised to integrate the complementary advantages of feature reconstruction-based and focused pixel recombination-based methods. Extensive experimental results demonstrate the effectiveness and superiority of the proposed method. The source code of the proposed method is available at: https://github.com/lhf12278/GRFusion.
BibTeX
@article{li2024generation,
  author = {Huafeng Li and Dan Wang and Yuxin Huang and Yafei Zhang and Zhengtao Yu},
  title = {{Generation and Recombination for Multifocus Image Fusion With Free Number of Inputs}},
  journal = {IEEE Trans. Circuits Syst. Video Technol.},
  year = {2024},
  url = {https://openreview.net/forum?id=MbKQCRsvYa},
}

Loading related content...

Figure