Joint Semi-supervised 3D Super-Resolution and Segmentation with Mixed Adversarial Gaussian Domain Adaptation

Nicoló Savioli, Antonio de Marvao, Wenjia Bai, Shuo Wang, Stuart A. Cook, Calvin W. L. Chin, Daniel Rueckert, Declan P. O'ReganarXiv 2021 · arXiv

Introduction

Accurate 3D reconstruction and segmentation of cardiac structures are critical for diagnosing cardiovascular diseases and conducting large-scale population studies. However, a significant gap exists between high-resolution (HR) 3D imaging capabilities and the "multi-slice 2D" scans common in clinical practice. These standard scans often suffer from low through-plane resolution and respiratory motion artifacts. Furthermore, deep learning models trained on specific cohorts often fail when applied to diverse ethnic populations due to domain shift. This paper presents a novel framework that addresses these challenges through joint multi-task learning and variational domain adaptation.

The Gemini-GAN Architecture

To bridge the gap between image enhancement and anatomical labeling, the authors introduced Gemini-GAN, a multi-task Generative Adversarial Network. Unlike traditional pipelines that treat Super-Resolution (SR) and Segmentation as sequential steps, Gemini-GAN performs them simultaneously.

The Gemini-GAN architecture utilizes a shared encoder and dual decoder branches for joint super-resolution and segmentation.

The architecture employs a single encoder to extract features from low-resolution (LR) input volumes. These features are then processed by two parallel decoder branches: one for reconstructing high-resolution grayscale images and another for generating 3D segmentation maps. A shared discriminator ensures that both outputs maintain high fidelity relative to real HR data. This joint approach allows the segmentation branch to act as a structural prior for the SR branch, while the SR branch provides the fine-grained details necessary for precise anatomical labeling.

Domain Adaptation via V-AMA

A primary hurdle in deploying cardiac AI is the variation in scanner hardware and anatomical proportions across different populations. To ensure robustness, the researchers developed the Variational Adversarial Mixture Autoencoder (V-AMA).

The V-AMA framework facilitates domain adaptation by mapping source data distributions to a target domain using variational inference.

V-AMA addresses the "mode collapse" common in standard GAN-based domain adaptation (like CycleGAN) by using variational inference. It maps the data distribution of a new "source" population (e.g., Asian) to the "target" distribution (e.g., European) used during initial training. By decoupling the transformation into two variational networks and utilizing a Gaussian mixture approach in the latent space, V-AMA preserves underlying anatomical content while adapting to new image styles.

Experimental Evaluation

The framework was validated using three distinct datasets:

  1. UK Digital Heart Project (UKDHP): 1,331 subjects used for primary training, providing ground truth HR/LR pairs.
  2. National Heart Center Singapore Biobank (SG): 205 subjects used to test cross-ethnic and cross-hardware generalization.
  3. UK Biobank (UKBB): 1,331 subjects used for qualitative "real-world" assessment where only LR data is available.

Key Results and Insights

The proposed method demonstrated superior performance across multiple metrics:

  • Segmentation Accuracy: Gemini-GAN achieved a Dice Index of 0.87 for the left ventricular cavity on the UKDHP dataset, significantly outperforming the previous state-of-the-art (0.82).
  • Image Quality: The model reached a Peak Signal-to-Noise Ratio (PSNR) of ~27 dB and a Structural Similarity Index (SSIM) of 0.88, providing much clearer reconstructions than standard interpolation.
  • Generalization: On the Singapore dataset, V-AMA maintained a Dice Index of 0.81, whereas competing models like CycleGAN and MUNIT dropped to 0.72 and 0.74, respectively.

The authors highlight that skip connections between the encoder and decoders are vital for preserving spatial information, which is essential for accurate 3D heart modeling.

Conclusion and Future Directions

This research provides a robust solution for enhancing clinical cardiac MRI scans. By integrating super-resolution, segmentation, and domain adaptation into a unified framework, the system produces high-fidelity 3D models that generalize across diverse populations. Future work will focus on automating the initial rigid alignment process using regression networks and conducting further expert audits on large-scale datasets like the UK Biobank.

Summary updated

Abstract

Optimising the analysis of cardiac structure and function requires accurate 3D representations of shape and motion. However, techniques such as cardiac magnetic resonance imaging are conventionally limited to acquiring contiguous cross-sectional slices with low through-plane resolution and potential inter-slice spatial misalignment. Super-resolution in medical imaging aims to increase the resolution of images but is conventionally trained on features from low resolution datasets and does not super-resolve corresponding segmentations. Here we propose a semi-supervised multi-task generative adversarial network (Gemini-GAN) that performs joint super-resolution of the images and their labels using a ground truth of high resolution 3D cines and segmentations, while an unsupervised variational adversarial mixture autoencoder (V-AMA) is used for continuous domain adaptation. Our proposed approach is extensively evaluated on two transnational multi-ethnic populations of 1,331 and 205 adults respectively, delivering an improvement on state of the art methods in terms of Dice index, peak signal to noise ratio, and structural similarity index measure. This framework also exceeds the performance of state of the art generative domain adaptation models on external validation (Dice index 0.81 vs 0.74 for the left ventricle). This demonstrates how joint super-resolution and segmentation, trained on 3D ground-truth data with cross-domain generalization, enables robust precision phenotyping in diverse populations.
BibTeX
@misc{savioli2021joint,
  author = {Nicoló Savioli and Antonio de Marvao and Wenjia Bai and Shuo Wang and Stuart A. Cook and Calvin W. L. Chin and Daniel Rueckert and Declan P. O'Regan},
  title = {{Joint Semi-supervised 3D Super-Resolution and Segmentation with Mixed Adversarial Gaussian Domain Adaptation}},
  year = {2021},
  eprint = {2107.07975},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2107.07975},
}
Figure