An Explainable Deep Framework: Towards Task-Specific Fusion for Multi-to-One MRI Synthesis

Tao Tan, Luyi Han, Tianyu Zhang, Yunzhi Huang, Haoran Dou, Xin Wang, Yuan Gao, Chunyao Lu, Tao Tan 0002, Ritse MannMICCAI 2023 · DOI

TL;DR

This paper introduces TSF-Seq2Seq, a novel deep learning framework for synthesizing missing MRI sequences. It addresses the critical need for explainability and reliability in MRI synthesis by quantifying the contribution of each input sequence and highlighting areas of refinement, leading to improved synthesis performance.

What Problem Does This Paper Address?

Magnetic Resonance Imaging (MRI) often relies on multiple sequences (e.g., T1, T1Gd, T2, Flair) to provide comprehensive information for diagnosis and treatment. However, these sequences can be missing or unusable due to various factors like technical issues, artifacts, or patient-specific constraints. MRI synthesis aims to generate these missing sequences from available ones. While deep learning methods have shown promise, existing approaches often lack transparency. They struggle to explain why a particular sequence was synthesized or how much each input contributed, making it difficult for clinicians to trust and utilize the generated images in critical medical decisions. This paper tackles this gap by proposing an "explainable" framework that not only synthesizes missing MRI sequences but also provides insights into its decision-making process.

Methods

The core of the proposed method is the TSF-Seq2Seq (Task-Specific Fusion Sequence-to-Sequence) network, built upon an autoencoder-like architecture with an encoder (E) and a decoder (G). The novelty lies in its multi-sequence fusion module and the Task-Specific Enhanced Map (TSEM).

The fusion module incorporates two key components:

  1. Task-Specific Weighted Average: This module learns to assign adaptive weights to different input sequences based on the specific synthesis task (i.e., which sequence to generate from which inputs). This is achieved using a trainable fully connected layer that predicts weights, allowing the network to quantify the contribution of each input sequence.
  2. Task-Specific Attention: This module further refines the fused features at a pixel level using a conditional attention mechanism. It helps to focus on relevant areas and improve the quality of the synthesized image.

The TSEM is a novel interpretability tool. It's generated by subtracting the synthesized image produced without the attention module from the one with it. This map highlights the areas where the attention mechanism made significant refinements, offering a clear interpretation of what the network focused on during synthesis and indicating potential areas of uncertainty or complexity.

Datasets

The experiments were conducted on the Brain Tumor Segmentation 2021 (BraTS2021) dataset. This dataset comprises 1,251 subjects with four aligned MRI sequences: T1, T1Gd, T2, and Flair. The dataset was split into training (830 subjects), validation (93 subjects), and testing (328 subjects). Images were normalized and cropped to 128x192x192. For training, a random subset of sequences was used as input, with the remaining as targets. For validation and testing, fixed input combinations and targets were used.

Key Insights and Ideas

  • Explainability is Crucial for Clinical Adoption: The paper strongly advocates that for deep learning models to be useful in clinical settings, they must not only be accurate but also interpretable and reliable.
  • Quantifying Input Contributions: The task-specific weighted average module provides a direct way to understand how much each available MRI sequence influences the synthesis of a target sequence. This is a significant step beyond black-box synthesis.
  • TSEM as a Reliable Uncertainty Map: The TSEM offers a higher-resolution and more interpretable visualization of the network's refinement process compared to traditional attention maps, potentially serving as an indicator of synthesis difficulty or uncertainty.
  • Task-Specific Adaptation: The framework's ability to adapt its fusion strategy based on the specific input-target combination is key to its flexibility and performance across various scenarios.

Key Results

The proposed TSF-Seq2Seq method demonstrated superior performance compared to state-of-the-art methods across various input combinations, as evidenced by higher PSNR and SSIM scores, and lower LPIPS scores (indicating better perceptual quality). For instance, in the case of synthesizing a target sequence from three inputs, TSF-Seq2Seq achieved a PSNR of 28.8±2.6, SSIM of 0.887±0.042, and LPIPS of 8.89±3.80, outperforming other methods. The ablation study confirmed that both the task-specific weighted average and the task-specific attention module contribute to the overall performance, with the attention module being particularly effective in refining fused features.

Limitations and Future Work

The authors acknowledge that the TSF-Seq2Seq framework is primarily designed for MRI synthesis. They suggest exploring its application in assisting downstream multi-sequence analysis tasks in future work. While the TSEM provides a valuable interpretability tool, its direct integration into clinical decision-making workflows would require further validation and standardization. The paper also mentions that for multi-input scenarios, one-to-one translation methods synthesize multiple outputs separately and then average them, which is a less integrated approach compared to the proposed fusion strategy.

Summary updated

BibTeX
@inproceedings{tan2023explainable,
  author = {Tao Tan and Luyi Han and Tianyu Zhang and Yunzhi Huang and Haoran Dou and Xin Wang and Yuan Gao and Chunyao Lu and Tao Tan and Ritse Mann},
  title = {{An Explainable Deep Framework: Towards Task-Specific Fusion for Multi-to-One MRI Synthesis}},
  booktitle = {International Conference on Medical Image Computing and Computer-Assisted Intervention},
  year = {2023},
  doi = {10.1007/978-3-031-43999-5\_5},
}
Figure