Abstract
In clinical practice, brain MRI datasets are frequently incomplete due to time constraints, motion artifacts, or patient discomfort. This paper introduces HF-GAN (Hybrid-Fusion GAN), a unified framework designed to synthesize any missing MRI sequence from an arbitrary subset of available inputs. By leveraging principled feature disentanglement, HF-GAN separates modality-specific structural information from the complex synergies between sequences. Experimental results on the BraTS 2018 and IXI datasets demonstrate that HF-GAN achieves state-of-the-art synthesis quality and significantly improves performance in downstream clinical tasks, such as tumor segmentation.
Introduction
Modern AI-driven diagnostic tools typically require a complete set of MRI sequences (e.g., T1, T2, T1c, and FLAIR) to function effectively. However, real-world clinical scenarios often result in missing sequences, rendering many automated analysis tools unusable. While deep learning has been applied to synthesize missing data, previous methods often struggled with arbitrary input combinations or failed to capture non-linear relationships between different MRI contrasts.
This research addresses these challenges by proposing a framework that can handle all $2^N - 2$ possible combinations of missing sequences within a single unified network, ensuring high-fidelity synthesis even when input data is sparse.
Methodology
The core innovation of HF-GAN lies in its principled feature disentanglement strategy, which aligns the neural network architecture with the varying complexity of medical imaging features.
Hybrid-Fusion Encoder ($Enc^{HF}$)
Instead of a monolithic encoder, HF-GAN employs a dual-pathway system:
- Modality-Specific Pathway: Parallel encoders extract fundamental structural features from each individual available sequence.
- Complementary Pathway: A multi-channel encoder identifies "synergistic" information—complex features that only emerge when multiple sequences are analyzed simultaneously, which is critical for accurate pathology detection.
Feature Fusion and Infusion
The Channel Attention-based Feature Fusion (CAFF) module acts as a traffic controller, weighting features by importance before projecting them into a common latent space ($z$). Subsequently, a transformer-based Modality Infuser (MI) takes this generic anatomical representation and "infuses" it with the specific characteristics of the target sequence ($t$) to be generated.
Intensity Encoding (IE) Module
To mitigate "stripe artifacts" common in 2D slice-based synthesis, the authors integrated a lightweight Intensity Encoding module. This provides a 3D intensity prior, ensuring volumetric consistency across the synthesized brain images without the heavy computational cost of fully 3D models.
Results and Discussion
The model was rigorously evaluated on two major benchmarks: BraTS 2018 (focused on glioblastoma) and the IXI Dataset (focused on healthy anatomy).
- State-of-the-Art Performance: HF-GAN achieved an average PSNR of 30.27 on the BraTS dataset, outperforming established models like ResViT and Hyper-GAE.
- Visual Fidelity: Qualitative analysis shows that HF-GAN produces sharper tumor boundaries and more accurate internal structures compared to competitors, which often produce blurred results when inputs are limited.
- Downstream Clinical Impact: The practical utility of the framework was demonstrated by using synthesized sequences to "fill the blanks" for a tumor segmentation model. This intervention improved the segmentation Dice score from a failing 0.38 to a highly usable 0.78.
Conclusion
HF-GAN represents a significant advancement in unified medical image synthesis. By prioritizing the disentanglement of complementary and modality-specific features, the framework provides a robust solution for handling missing clinical data. Future work may involve integrating these disentanglement principles into emerging architectures like Diffusion Models or State-Space Models to further enhance synthesis quality and efficiency.
Summary updated