Abstract
Multisequence Magnetic Resonance Imaging (MRI) provides a more reliable diagnosis in clinical applications through complementary information across sequences. However, in practice, the absence of certain MR sequences is a common problem that can lead to inconsistent analysis results. In this work, we propose a novel unified framework for synthesizing multisequence MR images, called hybrid-fusion GAN (HF-GAN). The fundamental mechanism of this work is principled feature disentanglement, which aligns the design of the architecture with the complexity of the features. A powerful many-to-one stream is constructed for the extraction of complex complementary features, while utilizing parallel, one-to-one streams to process modality-specific information. These disentangled features are dynamically integrated into a common latent space by a channel attention-based fusion module (CAFF) and then transformed via a modality infuser to generate the target sequence. We validated our framework on public datasets of both healthy and pathological brain MRI. Quantitative and qualitative results show that HF-GAN achieves state-of-the-art performance, with our 2D slice-based framework notably outperforming a leading 3D volumetric model. Furthermore, the utilization of HF-GAN for data imputation substantially improves the performance of the downstream brain tumor segmentation task, demonstrating its clinical relevance.
Introduction
Magnetic Resonance Imaging (MRI) is a cornerstone of modern clinical diagnostics, offering various sequences such as T1-weighted, T2-weighted, and FLAIR to highlight different tissue characteristics. In clinical practice, however, acquiring a complete set of sequences is often hindered by time constraints, patient discomfort, or motion artifacts. This missing data problem significantly impacts automated analysis tools, such as tumor segmentation algorithms, which typically require a full suite of sequences to function accurately.
While image synthesis methods have been proposed to "fill in" these missing sequences, existing approaches often struggle with arbitrary input combinations or fail to capture the intricate, non-linear relationships between different MRI contrasts. This paper introduces HF-GAN, a framework designed to overcome these limitations through principled feature disentanglement.
Methodology: The HF-GAN Framework
The core innovation of HF-GAN is its architecture, which explicitly separates anatomical features from modality-specific details. The framework consists of three primary components:
- Hybrid-Fusion Encoder ($Enc^{HF}$): Unlike traditional encoders that treat all inputs uniformly, HF-GAN employs a dual-stream approach. Modality-specific encoders ($Enc^M$) extract unique features from each available scan. Simultaneously, a "many-to-one" complementary encoder ($Enc^C$) focuses on the complex interactions between sequences, which is vital for accurately representing pathological regions like tumors.
- Channel Attention-based Feature Fusion (CAFF): This module dynamically weighs the importance of features from different streams. It projects these features into a "common latent space" ($z$), creating a modality-agnostic anatomical representation of the brain.
- Modality Infuser (MI): Utilizing a transformer-based architecture, the MI takes the anatomical map from the common latent space and "infuses" it with the specific characteristics of the target sequence ($t$), producing a target latent space ($z_t$) for the final decoder.
To ensure consistency across 2D slices and avoid the "stripe artifacts" common in slice-based methods, the authors also introduced an Intensity Encoding (IE) module. This module provides a global intensity prior, allowing the 2D model to achieve 3D-like consistency without the computational burden of full 3D convolutions.
Experimental Results
The HF-GAN framework was evaluated on two major benchmarks: the BraTS 2018 dataset (pathological brain MRI) and the IXI dataset (healthy subjects).
Quantitative Performance
On the BraTS dataset, HF-GAN achieved state-of-the-art results, with an average Peak Signal-to-Noise Ratio (PSNR) of 30.27 and a Structural Similarity Index (SSIM) of 0.9393. Notably, this 2D-based framework outperformed leading 3D volumetric models like Hyper-GAE, demonstrating the effectiveness of the IE module and the disentanglement strategy.
Clinical Impact on Downstream Tasks
The practical utility of HF-GAN was demonstrated through a brain tumor segmentation task using nnU-Net. When only a single T2 sequence was available, the segmentation performance (Dice score) was a mere 0.3865. By using HF-GAN to synthesize the missing sequences (T1, T1c, FLAIR), the Dice score improved dramatically to 0.7848. This nearly 40-point increase underscores the framework's potential to rescue clinical workflows hampered by incomplete data.
Conclusion and Future Directions
HF-GAN represents a significant advancement in unified brain MRI synthesis. By aligning architectural design with feature complexity, it achieves high-fidelity results that are both anatomically accurate and clinically useful.
Future work may explore weight-sharing strategies to reduce memory consumption when dealing with a larger number of modalities. Additionally, integrating the disentanglement principles into newer generative architectures, such as Diffusion models or State-Space Models, could further enhance synthesis quality. The framework's reliance on spatial registration also suggests a path toward developing methods that can handle misaligned or unpaired datasets.
Summary updated