Visualisation of Medical Image Fusion and Translation for Accurate Diagnosis of High Grade Gliomas

Nishant Kumar, Nico Hoffmann, Matthias Kirsch, Stefan Gumhold

TL;DR

Medical image fusion and translation are powerful AI tools for brain tumor diagnosis, but they often act as "black boxes" that neurosurgeons find difficult to trust. This paper introduces a novel visualization framework that generates confidence heat maps based on mutual information, allowing clinicians to see exactly how much information is transferred from source scans to the final result and identify potential AI "hallucinations."

What Problem Does This Paper Address?

When treating high-grade gliomas (aggressive brain tumors), neurosurgeons rely on a mix of anatomical data (from MRI) and functional data (from PET). Image Fusion merges these into one view, while Image Translation uses AI to "synthesize" a missing scan (e.g., creating a T1 MRI from a T2 MRI).

The problem is trust. Current AI methods—especially Deep Learning and Generative Adversarial Networks (GANs)—can be highly intricate and prone to "hallucinating" features that aren't actually there. Standard evaluation metrics provide a single numerical score for an entire image, which is useless to a surgeon who needs to know if a specific pixel representing a tumor boundary is reliable. There is currently no "gold standard" for what a perfect fused image should look like, making it hard to verify these outputs in a clinical setting.

Methods

The authors propose a visualization technique that quantifies the information transfer between a source-target image pair.

  1. Confidence Heat Maps: Instead of a global score, they compute a confidence score for every pixel. This is done by analyzing image patches (specifically $7 \times 7$ pixels) to capture local context.
  2. Statistical Modeling: They treat the source and target images as random variables. Since the joint probability distribution (how the images relate) is unknown, they estimate it using Fréchet inequalities and Hoeffding’s covariance identity to find the upper and lower bounds of how the pixels correlate.
  3. Normalized Mutual Information (NMI): They use a patch-level NMI formula to calculate the confidence score ($S_{X,Y}$). A value near 1 indicates high confidence that the information in the output image actually came from the source image.
  4. RGB Visualisation: To make this intuitive for doctors, they map these scores to colors. For example, in a fused MRI/PET image, magenta might represent regions where MRI data is dominant, while cyan represents PET data. This allows a surgeon to see the "provenance" of every feature in the image.

Datasets

The study utilized two primary datasets:

  • Harvard Whole Brain Atlas: Used for fusion experiments. It contains pre-registered MR-T2 and PET-FDG image pairs from patients with high-grade gliomas (Grade III astrocytoma and Grade IV GBM).
  • BRATS 2013: Used for translation experiments. This dataset contains MR-T2 Flair (source) and MR-T1 (target) images, which the authors used to test how well AI can synthesize one modality from another.

Key Insights and Ideas

  • Detecting Hallucinations: The tool is particularly good at spotting when a GAN "invents" a tumor. If a translation model adds a tumor feature that wasn't supported by the source data, the confidence map flags that area as low-trust.
  • L1 vs. GAN: Surprisingly, the authors found that simpler translation methods (using L1 loss) were often more reliable and had higher confidence scores than complex GANs (like CycleGAN), especially when the training data was limited.
  • Sensitivity to Noise: By intentionally adding noise (Gaussian, Poisson, blur) to the images, the authors proved that their confidence maps correctly reflect a loss of information transfer. This confirms the tool is a robust "BS detector" for degraded or poorly processed images.
  • Modality Dominance: The visualization reveals which source modality "won" in the fusion process. For instance, they found that the RPCNN method is excellent at preserving MRI details but less reliable for PET features.

Key Results

  • Fusion Performance: The analysis showed that the FunFuseAn algorithm performed best at preserving PET (functional) features, while RPCNN was superior for MRI (anatomical) features.
  • Translation Reliability: In tests with 0% tumor cases in the training set, CycleGAN showed widespread "blue" regions (indicating very low confidence), whereas L1-based methods maintained higher confidence (cyan regions).
  • Visual Evidence: The paper provides clear visual evidence that as the percentage of tumor cases in training data increases, the reliability of GAN-based translation fluctuates wildly, while simpler methods remain more stable.

Limitations and Future Work

The authors acknowledge a few areas for improvement:

  • Probability Estimation: The current method uses a discrete bivariate distribution. They plan to explore Kernel Density Estimation (KDE) in the future to create even more accurate probability models for the feature vectors.
  • Clinical Integration: While the math is sound, the next step is to get this tool into the hands of neurosurgeons to evaluate how it changes their pre-operative decision-making in real-world scenarios.
  • Dimensionality: The current work focuses on 2D slices; extending this to full 3D volumetric confidence maps would be a natural progression for surgical planning.

Summary updated

Abstract

The medical image fusion combines two or more modalities into a single view while medical image translation synthesizes new images and assists in data augmentation. Together, these methods help in faster diagnosis of high grade malignant gliomas. However, they might be untrustworthy due to which neurosurgeons demand a robust visualisation tool to verify the reliability of the fusion and translation results before they make pre-operative surgical decisions. In this paper, we propose a novel approach to compute a confidence heat map between the source-target image pair by estimating the information transfer from the source to the target image using the joint probability distribution of the two images. We evaluate several fusion and translation methods using our visualisation procedure and showcase its robustness in enabling neurosurgeons to make finer clinical decisions.
BibTeX
@inproceedings{kumar2020visualisation,
  author = {Nishant Kumar and Nico Hoffmann and Matthias Kirsch and Stefan Gumhold},
  title = {{Visualisation of Medical Image Fusion and Translation for Accurate Diagnosis of High Grade Gliomas}},
  booktitle = {ISBI},
  year = {2020},
  url = {https://openreview.net/forum?id=mo4swo-fZNB},
}

Loading related content...

Figure