Editorial: Multimodal brain image fusion: Methods, evaluations, and applications

Yü Liu, Jiayi Ma, Qiang Zhang, Wei Wei, Xun Chen, Zheng LiuFrontiers in Neuroscience 2023 · DOI · OpenAlex

Abstract

Multimodal medical imaging, including Computed Tomography (CT), Magnetic Resonance Imaging (MRI), and Positron Emission Tomography (PET), provides critical pathological information for diagnosing brain diseases like Alzheimer's, gliomas, and epilepsy. This editorial summarizes 12 research papers that advance the field of multimodal brain image fusion, focusing on registration techniques, deep-learning-based fusion methods, and clinical applications ranging from holographic neurosurgery to glaucoma diagnosis.

Introduction: The Challenge of Multimodal Data

In neuroimaging, different modalities offer distinct advantages: CT highlights bone structure, MRI provides exquisite soft-tissue contrast, and PET tracks metabolic activity. Combining these into a single informative view—image fusion—is essential for comprehensive diagnosis and treatment planning. However, this process faces three primary hurdles: precise spatial alignment (registration), the integration of salient features without artifacts (fusion), and the objective assessment of the resulting image quality (evaluation).

Methodological Innovations in Fusion and Registration

Recent research has introduced sophisticated algorithms to address the technical complexities of merging disparate data types:

  • Advanced Registration: Wang et al. utilized a Bounded Generalized Gaussian Mixture Model (BGGMM) to describe joint intensity distributions, improving pixel-level alignment accuracy through a maximum likelihood framework.
  • Interpretable Fusion: Another study by Wang et al. employed an auto-encoder framework with a group lasso penalty to achieve disentangled representations. By separating redundant and complementary features, the method enhances the interpretability and quality of the fused images.
  • Feature Extraction: Zhang et al. proposed using local extreme maps to guide image filters, allowing for multi-scale extraction of bright and dark features from multimodal sources to create more informative fused images.
  • Human-Centric Evaluation: Tang et al. addressed the lack of dedicated quality metrics by developing a conditional generative adversarial network (cGAN) guided by radiologists' Mean Opinion Scores (MOS), aligning automated assessment with clinical expertise.

Clinical and Functional Applications

The utility of fusion extends beyond image generation into high-level diagnostic and surgical tasks:

  • Glaucoma Diagnosis: Yi et al. integrated fundus and visual field images using a Vision Transformer (ViT) architecture, which captures global dependencies to improve diagnostic accuracy.
  • Brain-Computer Interfaces (BCIs): Li et al. demonstrated that early-stage fusion of EEG and fNIRS data significantly outperforms middle or late-stage fusion for classification tasks in hybrid BCIs.
  • Tumor Segmentation: Hybrid models combining Convolutional Neural Networks (CNNs) for local textures and Transformers for global context have shown superior performance in brain tumor segmentation, as explored by Liu et al. and Xu et al.
  • Holographic Neurosurgery: Chen et al. introduced mixed reality holographic navigation, allowing surgeons to visualize intracranial structures in real-time by fusing holographic data with the physical surgical field.

Joint Analysis and Signal Processing

Beyond static imaging, joint analysis of functional data provides deeper insights into neural engagement. Mononen et al. highlighted the task-dependent nature of MEG-fMRI correlations, suggesting that electromagnetic-hemodynamic patterns are more sensitive proxies for neural engagement than isolated measures. Additionally, Rudolf et al. introduced improved methods for correcting ballistocardiographic (BCG) artifacts in resting-state EEG, ensuring more reliable spectral analysis and fMRI integration.

Conclusion and Future Directions

While significant progress has been made, the field continues to evolve toward real-time processing for surgical applications and the standardization of fusion quality metrics. The integration of deep learning architectures like Transformers with traditional signal processing remains a promising frontier for improving clinical outcomes in neurology. The research presented here provides a roadmap for future studies in multimodal brain image fusion, emphasizing the need for both technical rigor and clinical relevance.

Summary updated

Abstract

Multimodal medical imaging is playing an increasingly critical role in the diagnosis and treatment of various brain diseases like glioma, Alzheimer, ischemic stroke, epilepsy, etc. Medical images with different modalities such as computed tomography (CT), magnetic resonance imaging (MRI) and positron emission tomography (PET) focus on different categories of pathological information. Medical image fusion aims to combine the complementary information captured by different imaging modalities for better disease diagnosis and treatment. In recent years, medical image fusion has emerged as a very active topic with various fusion methods being proposed. In addition, the performance evaluation and downstream applications of medical image fusion are also attracting more and more attention. This Research Topic focuses on reporting advanced studies related to multimodal brain image fusion, including image fusion methods, objective evaluation approaches and specific applications in clinical problems. Twelve of the sixteen articles submitted to this Research Topic were accepted for publication after a thorough peer-review process. A summary of the key research findings of these works are provided from three aspects as below.Multimodal brain image registration, fusion and fusion quality evaluation: Image registration is the prerequisite of many medical image processing tasks such as fusion and segmentation. Wang et al. proposed a medical image registration method based on the bounded generalized Gaussian mixture model (BGGMM), which can thoroughly describe the joint intensity vector distribution of the image pixels and highlight the details of the image. The mixture model is formulated based on a maximum likelihood framework, and is solved by an expectation-maximization algorithm. With regard to image fusion methods, Wang et al. presented a disentangled representation based multimodal brain image fusion method via group lasso penalty using an auto-encoder-based deep learning framework, aiming to fully exploit the redundancy and complement prior relationships among multimodal source images. In their method, a complementary group lasso penalty is designed to promote the disentanglement ability and ensure the complementary feature maps of significant modality information. This study demonstrated that the disentangled representations can improve the interpretability of feature representation, leading to better fusion quality. Zhang et al. proposed a local extreme map guided multimodal brain image fusion method to improve the feature extraction ability of the guided image filter. By iteratively applying this local extreme map guided image filter, the proposed method is able to extract multiple scales of bright and dark features from the multimodal brain images, and integrate these salient features into one informative fused image. In addition, the proposed scheme can be incorporated with various guided filters or other similar filters in pursuit of improving their feature extraction ability. In comparison to the great attention paid to the study of image fusion methods, few works have explored dedicated quality assessment approaches for medical image fusion. To address this issue, Tang et al. proposed a novel quality assessment method for medial image fusion based on the conditional generative adversarial networks with the mean opinion scores (MOS) of the radiologists as the guiding condition. Their experimental results show that the proposed method outperforms several commonly-used quality assessment metrics of image fusion, with excellent agreement with subjective evaluations.Applications of multimodal brain image fusion: Multimodal medical image fusion has verified to be of great significance in various related high-level vision tasks such as classification and segmentation. Yi et al. proposed a multimodal classification architecture for the severity diagnosis of glaucoma. The proposed method integrates fundus images and gray scale images of the visual field as the input of the classification model. In addition, they introduced a plug-and-play classifier that adopts the Vision Transformer to extract the global dependencies of images, leading to improved accuracy of the diagnostic task. Li et al. conducted a study to investigate the stage of bi-modal fusion based on EEG and fNIRS for the classification task in hybrid brain-computer interfaces (BCIs). A Y-shaped neural network that fuses the bi-modal information in different stages was proposed. This study demonstrated that the early-stage fusion of EEG and fNIRS have significantly higher performance compared to middle-stage fusion and late-stage fusion. Liu et al. introduced both pixel-level and feature-level multimodal image fusion technique for brain tumor segmentation, aiming to achieve more sufficient and finer utilization of multimodal information. They presented a convolutional neural network (CNN)-based 3D pixel-level image fusion network to enrich the input modalities of the segmentation model and designed an attention-based feature fusion module for multimodal feature refinement. Xu et al. proposed a hybrid feature extraction network for medical image segmentation based on CNNs and Transformer. The proposed network can integrate the advantages of Transformer in capturing global contextual information and CNNs in extracting local features. Additionally, a multi-dimensional statistical feature extraction module is designed to strengthen low-dimensional texture features and enhance the segmentation performance. Tian et al. presented a method to combine light sheet microscopy (LSM) data with magnetic resonance histology (MRH) of the same specimen to restore the morphology of the LSM images to the in-skull geometry. They developed an image processing pipeline to restore the correct brain morphology of 3-dimensional cleared, stained mouse brain by registering the cleared brain data to MRH of the same specimen. Chen et al. introduced the minimally invasive puncture and drainage (MIPD) surgery using mixed reality holographic navigation technology (MRHNT). In their scheme, the holographic image and real head are integrated. By wearing mixed reality holographic equipment, the precise location of intracranial hematomas, tumors, ventricles, and other structures with the perspective function can be understood, which laid a theoretical foundation for implementation in neurosurgery.Joint analysis of multimodal data: Mononen et al. conducted a study to evaluate the across-tasks variability of magnetoencephalography (MEG)-functional magnetic resonance imaging (fMRI) relationship using data recorded during three distinct naming tasks from the same set of participants. Experiments results demonstrated that the MEG-fMRI correlation pattern varies according to the performed task. In addition, the electromagnetic-hemodynamic correlation could serve as a more sensitive proxy for task-dependent neural engagement in cognitive tasks than isolated within-modality measures. Rudolf et al. characterized the impact of the ballistocardiographic (BCG) artifact on resting-state EEG spectral properties and compared the effectiveness of seven common BCG correction methods to preserve EEG spectral features. They also assessed if these methods retained posterior alpha power reactivity to an eyes closure-opening task and compared the results from EEG-informed fMRI analysis using different BCG correction approaches. We appreciate the contributions of all the authors, the reviewers, and the editorial board members for contributing to this Research Topic. We hope that this Research Topic can make a broader impact on the study of multimodal brain image fusion including methods, evaluations and applications.
BibTeX
@inproceedings{liu2023editorial,
  author = {Yü Liu and Jiayi Ma and Qiang Zhang and Wei Wei and Xun Chen and Zheng Liu},
  title = {{Editorial: Multimodal brain image fusion: Methods, evaluations, and applications}},
  booktitle = {Frontiers in Neuroscience},
  year = {2023},
  doi = {10.3389/fnins.2022.1128938},
}

Loading related content...

Figure