Adversarial Pulmonary Pathology Translation for Pairwise Chest X-Ray Data Augmentation

Yunyan Xing, Zongyuan Ge, Rui Zeng, Dwarikanath Mahapatra, Jarrel Seah, Meng Law, Tom DrummondMICCAI 2019 · DOI · OpenReview

Abstract: Recent works show that Generative Adversarial Networks (GANs) can be successfully applied to chest X-ray data augmentation for lung disease recognition. However, the implausible and distorted pathology features generated from the less than perfect generator may lead to wrong clinical decisions. Why not keep the original pathology region? We proposed a novel approach that allows our generative model to generate high quality plausible images that contain undistorted pathology areas. The main idea is to design a training scheme based on an image-to-image translation network to introduce variations of new lung features around the pathology ground-truth area. Moreover, our model is able to leverage both annotated disease images and unannotated healthy lung images for the purpose of generation. We demonstrate the effectiveness of our model on two tasks: (i) we invite certified radiologists to assess the quality of the generated synthetic images against real and other state-of-the-art generative models, and (ii) data augmentation to improve the performance of disease localisation.

Introduction

Deep learning models for medical imaging require massive amounts of high-quality, annotated data. However, medical datasets are often small, especially for rare diseases, and expert annotation—such as drawing bounding boxes around tumors—is expensive and time-consuming. While standard data augmentation techniques like rotation or flipping provide some benefit, researchers have recently turned to Generative Adversarial Networks (GANs) to synthesize entirely new medical images.

A significant hurdle in this domain is that most GANs (e.g., StarGAN) often distort the very pathology they are intended to replicate. In a clinical setting, a blurry or anatomically impossible tumor is useless for training. This paper addresses this by asking: instead of trying to generate a new pathology, why not keep the original, undistorted pathology and generate a new "background" lung around it?

Methodology

The researchers utilize a Pix2Pix (image-to-image translation) framework to implement Adversarial Pulmonary Pathology Translation. The core innovation lies in preserving the pathology pixels while hallucinating the surrounding anatomy.

Comparison of the baseline StarGAN approach and the proposed Pix2Pix-based pathology translation framework.

The Training Process

  1. Artificial Pairing: Since Pix2Pix requires paired "before" and "after" images, the authors created them artificially. They took real X-rays with bounding box annotations, masked out everything outside the box (creating a "pathology only" image), and set the original full image as the target.
  2. Inpainting Task: The generator is trained to take the isolated pathology box and reconstruct a plausible, healthy-looking lung structure around it. This forces the model to learn the global context of a chest X-ray while the ground-truth disease remains unchanged.
  3. Pix2Pix-N Variant: To improve diversity, the authors added 2,000 unannotated healthy lung images to the training set. By applying random masks to these healthy images, the model learned to reconstruct "normal" anatomy more robustly, a variant they termed Pix2Pix-N.

Datasets and Evaluation

The study utilized the NIH Chest X-ray dataset, focusing on a subset of 880 images containing 984 expert-labeled bounding boxes across six pathologies: Atelectasis, Cardiomegaly, Effusion, Infiltration, Pneumonia, and Pneumothorax.

Evaluation was conducted through two primary methods: a "Turing Test" with certified radiologists and a downstream disease localization task using a Faster R-CNN model.

Results and Discussion

Visual Fidelity and Expert Review

Two radiologists (one senior with 10 years of experience and one registrar) were asked to distinguish real images from synthetic ones. The results showed a stark difference between the baseline and the proposed method. While StarGAN images were easily identified as fake, the Pix2Pix-N images frequently fooled the experts. For instance, the senior radiologist identified 108 out of 150 synthetic Pix2Pix-N images as real, compared to only 15 for the StarGAN baseline.

Qualitative comparison of real chest X-rays and synthetic images generated by StarGAN, Pix2Pix, and Pix2Pix-N.

Impact on Disease Localization

When used for data augmentation, the synthetic images significantly boosted the performance of a Faster R-CNN model. Using the "Ori+Pix2Pix-N" dataset (original data plus synthetic data) improved the Total Correct Location (CL) accuracy from 0.328 to 0.405 at 15k training steps. The model trained with augmented data converged faster and achieved higher accuracy across almost all disease categories.

Conclusion

The core insight of this work is that GANs are currently more proficient at generating "normal" anatomy than complex, variable disease features. By "locking" the disease pixels and lowering the bar for the generator to focus on background reconstruction, the researchers achieved higher-quality outputs that are clinically relevant. Future work may explore scaling this inpainting approach to other modalities like CT or MRI and addressing potential mismatches between pathology and generated anatomical context.

Summary updated

Abstract

Recent works show that Generative Adversarial Networks (GANs) can be successfully applied to chest X-ray data augmentation for lung disease recognition. However, the implausible and distorted pathology features generated from the less than perfect generator may lead to wrong clinical decisions. Why not keep the original pathology region? We proposed a novel approach that allows our generative model to generate high quality plausible images that contain undistorted pathology areas. The main idea is to design a training scheme based on an image-to-image translation network to introduce variations of new lung features around the pathology ground-truth area. Moreover, our model is able to leverage both annotated disease images and unannotated healthy lung images for the purpose of generation. We demonstrate the effectiveness of our model on two tasks: (i) we invite certified radiologists to assess the quality of the generated synthetic images against real and other state-of-the-art generative models, and (ii) data augmentation to improve the performance of disease localisation.
BibTeX
@inproceedings{xing2019adversarial,
  author = {Yunyan Xing and Zongyuan Ge and Rui Zeng and Dwarikanath Mahapatra and Jarrel Seah and Meng Law and Tom Drummond},
  title = {{Adversarial Pulmonary Pathology Translation for Pairwise Chest X-Ray Data Augmentation}},
  booktitle = {International Conference on Medical Image Computing and Computer-Assisted Intervention},
  year = {2019},
  doi = {10.1007/978-3-030-32226-7\_84},
}
Figure