Non-adversarial Learning: Vector-Quantized Common Latent Space for Multi-sequence MRI

Tao Tan, Luyi Han, Tao Tan 0002, Tianyu Zhang, Xin Wang, Yuan Gao, Chunyao Lu, Xinglong Liang, Haoran Dou, Yunzhi Huang, Ritse Mann

TL;DR

This paper introduces VQ-Seq2Seq, a novel non-adversarial method for synthesizing missing MRI sequences. By learning a shared "vector-quantized common" (VQC) latent space, it achieves robust MRI synthesis that is less prone to common GAN training issues like instability and mode collapse, while also demonstrating impressive anti-interference capabilities against noise and artifacts.

What Problem Does This Paper Address?

Magnetic Resonance Imaging (MRI) often requires multiple sequences to provide a comprehensive view of tissues. However, in clinical practice, some sequences might be missing or unusable, necessitating rescanning or disrupting downstream analysis. While generative adversarial networks (GANs) can synthesize these missing sequences, they typically require large amounts of paired training data and are notorious for training instability and mode collapse, leading to poor image quality. Existing methods that try to overcome this by using intermediate sequences often fail to learn a truly common latent space, limiting their effectiveness. This paper aims to address these limitations by developing a robust, non-adversarial approach for multi-sequence MRI synthesis.

Methods

The core of the proposed VQ-Seq2Seq method lies in learning a Vector-Quantized Common (VQC) latent space. Here's a breakdown:

  • VQ-VAE for Discrete Representation: They first use a Vector Quantized Variational Autoencoder (VQ-VAE) to compress input MRI images into a discrete latent space. This discrete representation is beneficial as it can capture more structured features while ignoring irrelevant details like artifacts.
  • Estimating the VQC Latent Space: Instead of directly using the discrete latent codes, they estimate a Gaussian distribution for a "common" latent space that represents multiple sequences. This is achieved by analyzing the statistics of the VQ-VAE's discrete latent codes and incorporating an uncertainty estimation to account for sequence-specific information.
  • Dynamic Model (Seq2Seq) for Synthesis: A dynamic model, specifically a Seq2Seq architecture, is then employed. This model takes samples from the VQC latent space and generates the target MRI sequences. This architecture allows for integrating multiple generation tasks and utilizing intermediate sequences.
  • Latent Space Consistency: To ensure that representations from different sequences are well-aligned within the VQC space, they introduce a consistency loss. This loss combines Mean Squared Error (MSE) with a contrastive learning component to pull representations of the same underlying anatomy from different sequences closer together.
  • Domain Augmentation: To further improve model stability and anti-interference capabilities, they employ random domain augmentation techniques. This includes intensity transformations (gamma, noise, bias field) and cross-sequence translation, which helps the model generalize better to variations and imperfections in input data.

The key novelty lies in the non-adversarial approach and the explicit construction of a shared, probabilistic VQC latent space that bridges different MRI sequences, moving beyond simply disentangling features.

Datasets

The primary dataset used for training and evaluation is the Brain Tumor Segmentation 2021 (BraTS2021) dataset. This dataset contains 1,251 subjects with four aligned MRI sequences: T1, T1Gd, T2, and Flair. To simulate clinical scenarios with missing sequences, the training set was divided into subsets with paired sequences (e.g., T1 and T1Gd, T1Gd and T2, T2 and Flair), creating situations where direct paired data between certain sequences (like T1 and Flair) was unavailable.

Key Insights and Ideas

  • Non-Adversarial Synthesis is Viable and Effective: The paper demonstrates that high-quality MRI synthesis can be achieved without relying on GANs, circumventing common training challenges.
  • A Shared Latent Space is Crucial: By learning a common latent space (VQC), the model can effectively bridge the gap between different MRI sequences, enabling robust generation even when direct paired data is scarce.
  • Discrete Representations Aid Robustness: The use of VQ-VAE for discrete latent representations helps the model ignore noise and artifacts, leading to cleaner synthesized images.
  • Anti-Interference Capabilities: The VQC latent space exhibits remarkable resilience to input noise, bias fields, and artifacts, suggesting its potential for robust medical image analysis.
  • One-Shot Segmentation Potential: The strong semantic representation learned in the VQC latent space shows promise for few-shot or one-shot segmentation tasks, a significant advantage in data-scarce scenarios.

Key Results

  • Superior Synthesis Performance: VQ-Seq2Seq outperforms several state-of-the-art GAN-based methods (MM-GAN, ResViT, Jiang et al., Seq2Seq) in terms of PSNR, SSIM, and LPIPS metrics for multi-sequence MRI synthesis, especially in scenarios with unpaired data. For instance, in T1→Flair translation, VQ-Seq2Seq achieves competitive results even with single-step generation, where other methods see a sharp performance drop.
  • Robustness to Noise and Artifacts: Table 2 shows that VQ-Seq2Seq significantly outperforms other methods in reconstructing images from noisy or bias-field corrupted inputs, achieving higher PSNR and SSIM values. Figure 4 visually confirms its ability to remove artifacts.
  • Effective Latent Space Dimensions: Experiments show that a latent dimension D=3 and an embedding dimension K=256 provide optimal synthesis performance.
  • One-Shot Segmentation Success: When trained on the VQC latent space, the nnU-Net model for brain tumor segmentation achieves better performance than when trained on individual sequences alone, demonstrating the semantic richness of the learned space.

Limitations and Future Work

  • Computational Cost: While non-adversarial, the training process for VQ-Seq2Seq involves a large number of steps (1,000,000) and requires significant computational resources (NVIDIA GeForce RTX 3090 Ti GPU).
  • Parameter Tuning: The paper mentions that parameters like $\lambda_1, \lambda_2, \lambda_3$ for reconstruction loss and $A_{con}, A_{vq}$ for total loss are experimentally set. Further exploration into optimal hyperparameter tuning could be beneficial.
  • Future Work: The authors highlight the potential of the VQC latent space for one-shot segmentation. Future work could focus on further developing and evaluating this capability, as well as exploring its application to other downstream tasks in medical imaging. The paper also suggests that the learned latent space could be used for other applications beyond synthesis, such as anomaly detection or cross-modality registration.

Summary updated

Abstract

Adversarial learning helps generative models translate MRI from source to target sequence when lacking paired samples. However, implementing MRI synthesis with adversarial learning in clinical settings is challenging due to training instability and mode collapse. To address this issue, we leverage intermediate sequences to estimate the common latent space among multi-sequence MRI, enabling the reconstruction of distinct sequences from the common latent space. We propose a generative model that compresses discrete representations of each sequence to estimate the Gaussian distribution of vector-quantized common (VQC) latent space between multiple sequences. Moreover, we improve the latent space consistency with contrastive learning and increase model stability by domain augmentation. Experiments using BraTS2021 dataset show that our non-adversarial model outperforms other GAN-based methods, and VQC latent space aids our model to achieve (1) anti-interference ability, which can eliminate the effects of noise, bias fields, and artifacts, and (2) solid semantic representation ability, with the potential of one-shot segmentation. Our code is publicly available.
BibTeX
@inproceedings{tan2024non,
  author = {Tao Tan and Luyi Han and Tao Tan and Tianyu Zhang and Xin Wang and Yuan Gao and Chunyao Lu and Xinglong Liang and Haoran Dou and Yunzhi Huang and Ritse Mann},
  title = {{Non-adversarial Learning: Vector-Quantized Common Latent Space for Multi-sequence MRI}},
  booktitle = {International Conference on Medical Image Computing and Computer-Assisted Intervention},
  year = {2024},
  doi = {10.1007/978-3-031-72120-5\_45},
}
Figure