Multi-Scale Pixel-Based Image Fusion Using Multivariate Empirical Mode Decomposition

Naveed ur Rehman, Shoaib Ehsan, Syed Muhammad Umer Abdullah, Muhammad Jehanzaib Akhtar, Danilo P. Mandic, Klaus D. McDonald-MaierSensors 2015 · DOI

TL;DR

This paper introduces a novel image fusion technique using Multivariate Empirical Mode Decomposition (MEMD) to combine multiple images into a single, high-quality image. MEMD effectively addresses limitations of previous methods like mode mixing and misalignment, leading to improved fusion results for multi-exposure and multi-focus images.

What Problem Does This Paper Address?

Capturing a complete scene in a single image is often impossible due to limitations like depth of field (leading to out-of-focus areas) or varying lighting conditions (causing over- or under-exposure). Image fusion aims to overcome these limitations by combining information from multiple images to create a single, more informative image. Traditional fusion methods often rely on spatial or frequency domain transformations. While Empirical Mode Decomposition (EMD) offers a data-driven approach, its univariate nature struggles with multi-channel image data, leading to issues like mode mixing (where different scales are mixed within a single component) and mode misalignment (where components representing the same scale are not aligned across different images). This makes it difficult to accurately compare and fuse information at pixel and scale levels.

Methods

The core of this paper is the application of Multivariate Empirical Mode Decomposition (MEMD) for image fusion. MEMD is an extension of EMD designed for multivariate data, allowing it to process multiple input channels (images) simultaneously. Unlike univariate EMD, MEMD operates in multidimensional spaces, inherently aligning common frequency scales across different input images. This alignment is crucial for pixel-level fusion.

The proposed fusion algorithm follows these steps:

  1. MEMD Decomposition: Input images are converted into a multivariate signal and decomposed into multiple Intrinsic Mode Functions (IMFs) using MEMD. Each IMF represents a specific scale or frequency component.
  2. Activity Level Computation: For each IMF of each input image, a "local activity level" is computed within a sliding window. This quantifies the information content at that scale. Two methods are proposed: variance estimation and weighted averaging of pixel intensities.
  3. Weight Calculation: Local weighting factors are calculated for each IMF based on its activity level. IMFs with higher activity receive greater weight.
  4. Weighted Fusion: The IMFs from all input images at a given scale are multiplied by their respective weights and summed to create a fused IMF for that scale.
  5. Final Fusion: All fused IMFs are summed to produce the final fused image.

This approach is inherently multi-scale, performing pixel-based fusion at each scale (IMF) before combining them.

Datasets

The authors evaluated their method on two types of datasets:

  • Multi-exposure images: A dataset of 60 scenes from the DTU robot dataset [36], with seven input images per scene captured at different positions/exposures. A ground truth image was available for this dataset.
  • Multi-focus images: A dataset of 60 scenes, with seven input images per scene where different parts of the scene were out of focus. For these sets, a ground truth image was not available.

Key Insights and Ideas

  • MEMD's Advantage in Alignment: The key insight is that MEMD's inherent ability to align common frequency scales across multiple input channels directly addresses the mode misalignment problem in EMD-based fusion. This allows for meaningful pixel-level comparisons and fusion at each scale.
  • Overcoming Mode Mixing: By processing data collectively in multidimensional spaces, MEMD significantly reduces mode mixing, ensuring that each IMF represents a more distinct scale.
  • Data Adaptivity: Like EMD, MEMD is data-driven, meaning it doesn't rely on predefined basis functions or filter banks, making it adaptable to various signal characteristics.
  • Multi-Scale Fusion: The proposed method naturally lends itself to multi-scale fusion by applying the fusion rule to each IMF independently and then summing them.

Key Results

The proposed MEMD-based fusion method was compared against standard techniques like Principal Component Analysis (PCA), Discrete Wavelet Transform (DWT), and Non-Subsampled Contourlet Transform (NCT) (both single and multi-directional variants).

Qualitative Evaluation: * Multi-exposure: MEMD-based fused images were observed to be better illuminated and more focused, especially in challenging areas, compared to PCA and DWT. NCT-N showed some color artifacts, while MEMD provided a good balance of illumination and sharpness. * Multi-focus: MEMD-based fusion excelled in bringing near-camera objects into focus, outperforming PCA and DWT. While NCT-N also produced high-quality images and slightly better focus on near objects, MEMD was generally competitive.

Quantitative Evaluation: * Multi-exposure: MEMD consistently outperformed PCA and DWT across various metrics (Entropy, Spatial Frequency, QAB/F, UIQI). It also performed comparably to NCT-S and NCT-N, with statistical tests (McNemar's) showing no significant difference between MEMD and NCT-N, and MEMD being statistically better than PCA and DWT. * Multi-focus: MEMD showed significantly better results than PCA. While NCT-based methods performed slightly better in some metrics (QAB/F, PM), statistical tests indicated no significant difference between MEMD and NCT-S/NCT-N.

Limitations and Future Work

  • Computational Complexity: The authors acknowledge that MEMD computation is inherently more computationally expensive than PCA, DWT, and NCT.
  • Future Improvements: They suggest that algorithmic improvements and optimized implementations for MEMD could reduce computational cost. Additionally, a pre-processing step for image denoising before fusion could be beneficial.

Summary updated

BibTeX
@article{rehman2015multi,
  author = {Naveed ur Rehman and Shoaib Ehsan and Syed Muhammad Umer Abdullah and Muhammad Jehanzaib Akhtar and Danilo P. Mandic and Klaus D. McDonald-Maier},
  title = {{Multi-Scale Pixel-Based Image Fusion Using Multivariate Empirical Mode Decomposition}},
  journal = {Sensors},
  year = {2015},
  doi = {10.3390/s150510923},
}
Figure