Lung Swapping Autoencoder: Learning a Disentangled Structure-texture Representation of Chest Radiographs

Lei Zhou, Joseph Bae, Huidong Liu, Gagandeep Singh, Jeremy Green, Amit Gupta, Dimitris Samaras, Prateek PrasannaarXiv 2022 · arXiv

Summary

Researchers have developed the Lung Swapping Autoencoder (LSAE), a generative framework that disentangles a patient's anatomical structure from the lung tissue texture in chest X-rays. By "swapping" textures between images, the model learns a highly efficient representation of pulmonary disease that outperforms much larger models and enables a novel form of data augmentation for COVID-19 outcome prediction.

Introduction

In medical imaging, well-labeled datasets are critical for training AI models, yet they are expensive and time-consuming to produce. While self-supervised learning has advanced significantly in natural image processing, it often fails to incorporate domain-specific priors essential for medical diagnosis.

In chest radiographs (CXRs), many pathologies—including COVID-19—manifest as changes in texture (such as ground-glass opacities) rather than changes in structure (the shape of the ribs or heart). Standard autoencoders often "leak" information between these two factors, leading to blurred images or distorted anatomy. The LSAE aims to perfectly separate "how the lung looks" (pathology) from "where the lung is" (anatomy) to improve diagnostic accuracy and model interpretability.

Methodology

The LSAE architecture builds upon the Swapping Autoencoder (SAE) framework, which encodes an image into two distinct latent spaces: structure and texture. To train the model, the structure code of one image (e.g., $I_1$) is combined with the texture code of another (e.g., $I_2$) to generate a "hybrid" radiograph.

To optimize this for CXRs, the authors introduced two key innovations:

  1. In-lung Texture Supervision: The model utilizes a lung segmentation mask to ensure the texture encoder focuses exclusively on patches inside the lung fields, preventing the representation from being diluted by non-diagnostic background information.
  2. Out-of-lung Structural Distortion Suppression: A patchwise contrastive loss is implemented to force the hybrid image to maintain the exact local structural details of the original anatomical template in areas outside the lungs, preventing common generative artifacts like "ghosting."
Visual demonstration of disentangled structure and texture swapping in chest X-rays.

Datasets

The study utilized two primary datasets for evaluation:

  • ChestX-ray14: A massive public dataset containing 112,120 frontal-view CXRs from over 30,000 patients, used for large-scale pre-training and benchmarking.
  • COVOC: A multi-institutional COVID-19 dataset (N=393) curated by the authors, which includes labels for clinical outcomes such as the requirement for mechanical ventilation.

Key Results and Discussion

Efficiency and Diagnostic Signal

The authors hypothesize that for pulmonary disease, the "texture" latent code is the primary carrier of diagnostic information. Their results support this: the LSAE texture encoder uses only 5 million parameters—77% smaller than a standard Inception v3 model—yet achieved superior results. On the ChestX-ray14 dataset, the LSAE reached a mean AUC of 0.790, outperforming larger supervised models like ResNet-50.

COVID-19 Prognosis and Generalization

On the COVOC dataset, the LSAE reduced the error rate for predicting ventilation requirements by 13% compared to a strong Inception v3 baseline. Furthermore, the model's ability to swap textures enables a novel form of data augmentation. By transplanting disease textures onto healthy anatomical templates from different hospitals, the LSAE creates realistic, labeled synthetic data that helps the model generalize to new clinical environments.

Radiologist Validation

In a visual assessment, professional radiologists misidentified synthetic hybrid images as "real" 56% of the time, and hybrid patches as "real" 74% of the time. This high level of realism confirms that the model successfully captures and replicates pathological textures without introducing obvious artifacts.

Conclusion

The Lung Swapping Autoencoder provides a powerful framework for learning disentangled representations in medical imaging. By focusing on the textural signals that carry diagnostic information, the LSAE achieves state-of-the-art performance with a compact architecture and enables effective cross-institutional data augmentation. Future work will explore applying this framework to 3D imaging and rare diseases where data scarcity is a primary challenge.

Summary updated

Abstract

Well-labeled datasets of chest radiographs (CXRs) are difficult to acquire due to the high cost of annotation. Thus, it is desirable to learn a robust and transferable representation in an unsupervised manner to benefit tasks that lack labeled data. Unlike natural images, medical images have their own domain prior; e.g., we observe that many pulmonary diseases, such as the COVID-19, manifest as changes in the lung tissue texture rather than the anatomical structure. Therefore, we hypothesize that studying only the texture without the influence of structure variations would be advantageous for downstream prognostic and predictive modeling tasks. In this paper, we propose a generative framework, the Lung Swapping Autoencoder (LSAE), that learns factorized representations of a CXR to disentangle the texture factor from the structure factor. Specifically, by adversarial training, the LSAE is optimized to generate a hybrid image that preserves the lung shape in one image but inherits the lung texture of another. To demonstrate the effectiveness of the disentangled texture representation, we evaluate the texture encoder $Enc^t$ in LSAE on ChestX-ray14 (N=112,120), and our own multi-institutional COVID-19 outcome prediction dataset, COVOC (N=340 (Subset-1) + 53 (Subset-2)). On both datasets, we reach or surpass the state-of-the-art by finetuning $Enc^t$ in LSAE that is 77% smaller than a baseline Inception v3. Additionally, in semi-and-self supervised settings with a similar model budget, $Enc^t$ in LSAE is also competitive with the state-of-the-art MoCo. By "re-mixing" the texture and shape factors, we generate meaningful hybrid images that can augment the training set. This data augmentation method can further improve COVOC prediction performance. The improvement is consistent even when we directly evaluate the Subset-1 trained model on Subset-2 without any fine-tuning.
BibTeX
@misc{zhou2022lung,
  author = {Lei Zhou and Joseph Bae and Huidong Liu and Gagandeep Singh and Jeremy Green and Amit Gupta and Dimitris Samaras and Prateek Prasanna},
  title = {{Lung Swapping Autoencoder: Learning a Disentangled Structure-texture Representation of Chest Radiographs}},
  year = {2022},
  eprint = {2201.07344},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2201.07344},
}
Figure