Visualisation of Medical Image Fusion and Translation for Accurate Diagnosis of High Grade Gliomas

Nishant Kumar, Nico Hoffmann, Matthias Kirsch, Stefan Gumhold

Abstract

The medical image fusion combines two or more modalities into a single view while medical image translation synthesizes new images and assists in data augmentation. Together, these methods help in faster diagnosis of high grade malignant gliomas. However, they might be untrustworthy due to which neurosurgeons demand a robust visualisation tool to verify the reliability of the fusion and translation results before they make pre-operative surgical decisions. In this paper, we propose a novel approach to compute a confidence heat map between the source-target image pair by estimating the information transfer from the source to the target image using the joint probability distribution of the two images. We evaluate several fusion and translation methods using our visualisation procedure and showcase its robustness in enabling neurosurgeons to make finer clinical decisions.

Introduction

In the treatment of high-grade gliomas—aggressive brain tumors—neurosurgeons rely heavily on a combination of anatomical data from MRI and functional data from PET scans. Image Fusion integrates these modalities into a single comprehensive view, while Image Translation leverages AI to synthesize missing scans, such as generating a T1 MRI from a T2 source.

Despite their potential, these AI-driven methods often operate as "black boxes." Deep Learning and Generative Adversarial Networks (GANs) are prone to "hallucinating" features that lack a basis in the source data. Standard evaluation metrics typically provide a global numerical score, which fails to inform a surgeon about the reliability of specific regions, such as tumor boundaries. This paper addresses the critical need for trust by introducing a visualization framework that generates pixel-level confidence heat maps.

Proposed Methodology

The core of the proposed approach is the quantification of information transfer between source and target image pairs.

The proposed framework generates confidence maps for input images to create a final visualization that flags unreliable regions.
  1. Confidence Heat Maps: Instead of a single global score, the system computes a confidence value for every pixel by analyzing $7 \times 7$ patches to capture local context.
  2. Statistical Modeling: The images are treated as random variables. The joint probability distribution is estimated using Fréchet inequalities and Hoeffding’s covariance identity to establish bounds on pixel correlation.
  3. Normalized Mutual Information (NMI): A patch-level NMI formula calculates the confidence score ($S_{X,Y}$). A value near 1 indicates that the information in the output image is reliably derived from the source.
  4. RGB Visualisation: These scores are mapped to an intuitive RGB space. In a fused MRI/PET image, specific colors represent the dominance of one modality over the other (e.g., magenta for MRI, cyan for PET), allowing surgeons to trace the "provenance" of every feature.

Experimental Evaluation

The study utilized the Harvard Whole Brain Atlas for fusion experiments (MR-T2 and PET-FDG pairs) and the BRATS 2013 dataset for translation experiments (MR-T2 Flair to MR-T1 synthesis).

Fusion and Noise Robustness

The framework was used to evaluate several fusion algorithms, including FunFuseAn, LPCNN, and RPCNN. Results indicated that FunFuseAn excelled at preserving functional PET features, while RPCNN was superior for anatomical MRI details.

Comparison of fusion methods and their corresponding confidence maps. The bottom rows demonstrate how the confidence maps respond to various types of image noise.

To test the system's robustness as a "BS detector," the authors introduced various types of noise (Gaussian, Poisson, blur). The confidence maps correctly reflected a significant loss of information transfer in these degraded images, proving the tool's ability to flag unreliable data.

Translation and Hallucination Detection

A key finding was the tool's ability to spot AI "hallucinations." When GANs synthesized tumor features not supported by source data, the confidence maps flagged those areas as low-trust. Interestingly, simpler translation methods using L1 loss often proved more reliable than complex GANs like CycleGAN, particularly when training data was limited.

Conclusion

This visualization framework provides a necessary layer of transparency for AI in medical imaging. By moving beyond global metrics to local confidence maps, it empowers neurosurgeons to verify the reliability of fused and translated images. Future work will focus on improving probability estimation through Kernel Density Estimation (KDE) and extending the framework to 3D volumetric data for more comprehensive surgical planning.

Summary updated

Abstract

The medical image fusion combines two or more modalities into a single view while medical image translation synthesizes new images and assists in data augmentation. Together, these methods help in faster diagnosis of high grade malignant gliomas. However, they might be untrustworthy due to which neurosurgeons demand a robust visualisation tool to verify the reliability of the fusion and translation results before they make pre-operative surgical decisions. In this paper, we propose a novel approach to compute a confidence heat map between the source-target image pair by estimating the information transfer from the source to the target image using the joint probability distribution of the two images. We evaluate several fusion and translation methods using our visualisation procedure and showcase its robustness in enabling neurosurgeons to make finer clinical decisions.
BibTeX
@inproceedings{kumar2020visualisation,
  author = {Nishant Kumar and Nico Hoffmann and Matthias Kirsch and Stefan Gumhold},
  title = {{Visualisation of Medical Image Fusion and Translation for Accurate Diagnosis of High Grade Gliomas}},
  booktitle = {ISBI},
  year = {2020},
  url = {https://openreview.net/forum?id=mo4swo-fZNB},
}
Figure