Less Is More: Unsupervised Mask-Guided Annotated CT Image Synthesis With Minimum Manual Segmentations

Xiaodan Xing, Giorgos Papanastasiou, Simon Walsh, Guang Yang

Introduction

The development of robust deep learning models for medical imaging is frequently bottlenecked by the scarcity of high-quality, expert-labeled datasets. While Generative Adversarial Networks (GANs) have emerged as a powerful tool for data augmentation through synthetic image generation, standard Mask-to-Image (M2I) frameworks face significant hurdles. These include a heavy reliance on large volumes of paired training data, the tendency to "hallucinate" unrealistic anatomical structures outside the region of interest, and "mode collapse," where the generator produces repetitive, low-diversity outputs.

To address these challenges, this paper introduces UM2I (Unsupervised Mask-to-Image), a novel framework designed to synthesize high-fidelity CT images and corresponding annotations using only a fraction of the manual labels typically required.

Methodology

The UM2I framework operates through a three-stage pipeline designed to maximize structural integrity and label efficiency.

1. Superpixel-based Guidance

Rather than requiring manual masks for every training image, the system utilizes the Simple Linear Iterative Clustering (SLIC) algorithm. SLIC groups pixels into "superpixels" based on intensity values, creating a structural map of the entire CT slice—including bone, soft tissue, and air—without human intervention. This provides a "global" roadmap that prevents the GAN from hallucinating impossible anatomy.

2. Semi-Supervised Multi-task Learning

The core architecture is a StyleGAN-based generator modified for multi-task learning. The model is trained to simultaneously synthesize the CT image and its corresponding segmentation mask. By sharing features between these two tasks, the model develops a deeper semantic understanding of medical structures. A small subset of manually labeled data is used to "anchor" these unsupervised masks to clinical categories, allowing the model to generalize from minimal supervision.

3. Specialized Evaluation Metrics

Standard metrics like the Fréchet Inception Distance (FID) are often insufficient for medical imaging. The authors propose Multi-scale Multi-task versions (MM-FID and MM-STD) to specifically measure the fidelity and diversity of synthesized CT scans across various resolutions and anatomical layouts.

Experimental Setup

The framework was validated using the National COVID-19 Chest Image Database (NCCID), comprising 3,292 CT scans from 1,072 patients across 11 UK hospitals. To maintain high resolution (1024x1024) while managing GPU memory, the authors implemented a "CT montage" approach, tiling four axial slices into a single frame. The model's performance was tested across three data scenarios: Insufficient (300 labels), Moderate (600 labels), and Sufficient (1,500 labels).

Key Insights and Results

Structural Fidelity and Intensity Mapping

By using intensity-based superpixels, the GAN is forced to respect the physical properties of CT scans (e.g., Hounsfield Units). This is particularly effective for capturing Ground Glass Opacities (GGO)—the hazy, faint textures characteristic of COVID-19—which traditional GANs often fail to replicate accurately.

Performance Gains

The UM2I framework demonstrated superior utility in downstream tasks. In the "Insufficient data" scenario (300 labels), augmenting the training set with UM2I-generated images improved the lesion segmentation Dice score from 0.45 to 0.51. Furthermore, the MM-STD metric confirmed that UM2I produces a significantly wider variety of lesion shapes and image layouts compared to standard M2I methods, effectively mitigating mode collapse.

Conclusion and Future Work

UM2I demonstrates that unsupervised structural guidance can significantly reduce the labeling burden for medical AI. While the current 2D montage approach is highly efficient, future work will focus on extending this intensity-guided synthesis to 3D volumes. Additionally, the authors suggest that the principles of UM2I could be integrated into emerging generative architectures, such as Diffusion Models, to further enhance the realism and diversity of synthetic medical data.

Summary updated

Abstract

As a pragmatic data augmentation tool, data synthesis has generally returned dividends in performance for deep learning based medical image analysis. However, generating corresponding segmentation masks for synthetic medical images is laborious and subjective. To obtain paired synthetic medical images and segmentations, conditional generative models that use segmentation masks as synthesis conditions were proposed. However, these segmentation mask-conditioned generative models still relied on large, varied, and labeled training datasets, and they could only provide limited constraints on human anatomical structures, leading to unrealistic image features. Moreover, the invariant pixel-level conditions could reduce the variety of synthetic lesions and thus reduce the efficacy of data augmentation. To address these issues, in this work, we propose a novel strategy for medical image synthesis, namely Unsupervised Mask (UM)-guided synthesis, to obtain both synthetic images and segmentations using limited manual segmentation labels. We first develop a superpixel based algorithm to generate unsupervised structural guidance and then design a conditional generative model to synthesize images and annotations simultaneously from those unsupervised masks in a semi-supervised multi-task setting. In addition, we devise a multi-scale multi-task Fréchet Inception Distance (MM-FID) and multi-scale multi-task standard deviation (MM-STD) to harness both fidelity and variety evaluations of synthetic CT images. With multiple analyses on different scales, we could produce stable image quality measurements with high reproducibility. Compared with the segmentation mask guided synthesis, our UM-guided synthesis provided high-quality synthetic images with significantly higher fidelity, variety, and utility ( <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">${p}&lt; {0}.{05}$ </tex-math></inline-formula> by Wilcoxon Signed Ranked test).
BibTeX
@article{xing2023less,
  author = {Xiaodan Xing and Giorgos Papanastasiou and Simon Walsh and Guang Yang},
  title = {{Less Is More: Unsupervised Mask-Guided Annotated CT Image Synthesis With Minimum Manual Segmentations}},
  journal = {IEEE Trans. Medical Imaging},
  year = {2023},
  url = {https://openreview.net/forum?id=rKRVY9wTz6},
}
Figure