Task Driven Generative Modeling for Unsupervised Domain Adaptation: Application to X-ray Image Segmentation

Yue Zhang, Shun Miao, Tommaso Mansi, Rui Liao

TL;DR

This paper introduces TD-GAN (Task Driven Generative Adversarial Network), a framework that allows deep learning models to segment organs in real X-ray images without requiring a single manual label for those images. By training on synthetic X-rays derived from 3D CT scans and using a "task-aware" style transfer, the authors bridge the domain gap between simulation and reality, achieving segmentation accuracy nearly on par with fully supervised models.

What Problem Does This Paper Address?

Automatic segmentation (parsing) of organs in X-ray images is vital for guided surgery and diagnostic workflows. However, training these models is a "chicken and egg" problem:

  1. Labeling is hard: X-rays are 2D projections where organs overlap, making manual pixel-wise labeling by clinicians extremely difficult and time-consuming.
  2. Simulation is imperfect: While we can easily generate synthetic X-rays (called Digitally Reconstructed Radiographs, or DRRs) from labeled 3D CT scans, models trained on these "perfect" simulations usually fail when applied to real-world X-rays due to differences in texture, noise, and contrast (the domain gap).

The authors tackle Unsupervised Domain Adaptation: how to take a model learned in a labeled "source" domain (synthetic DRRs) and make it work in an unlabeled "target" domain (real X-rays).

Methods

The authors propose a two-stage pipeline:

  1. The Segmenter (DI2I): They first train a Dense Image-to-Image network, an encoder-decoder architecture similar to a U-Net but utilizing "Dense Blocks" (from DenseNet) to improve feature extraction and gradient flow. This model is trained to segment four organs (lung, heart, liver, and bone) using only synthetic DRRs and their perfectly known labels.
  2. The Bridge (TD-GAN): To handle real X-rays, they wrap the DI2I inside a Task Driven Generative Adversarial Network.
  3. Style Transfer: Based on CycleGAN, it uses two generators to translate images back and forth between the "DRR style" and "X-ray style."
    • Task-Driven Supervision: Unlike a standard CycleGAN, which only cares if an image looks realistic, TD-GAN's discriminator ($D_2$) is "task-driven." It evaluates whether a generated DRR (translated from a real X-ray) is actually segmentable by the pre-trained DI2I.
  4. Consistency Losses: They introduce a "Cycle Segmentation Consistency" loss, ensuring that if you translate a DRR to an X-ray and back, the final image still yields the same segmentation results as the original.

Datasets

  • Source Domain: 815 Digitally Reconstructed Radiographs (DRRs) generated from 3D CT volumes. These come with "ground truth" labels automatically derived from the CT segmentations.
  • Target Domain: 153 real "topograms" (scout X-ray views taken before a CT scan).
  • 73 topograms were used for unlabeled training.
  • 20 for validation.
  • 60 for testing.
  • Note: While the topograms had labels for evaluation purposes, the TD-GAN model never saw these labels during training.

Key Insights and Ideas

  • Anatomy over Aesthetics: Standard GANs often focus on global style (texture/brightness). By feeding the segmentation probability maps into the discriminator, the authors force the GAN to prioritize preserving anatomical structures (edges, shapes) that are critical for the segmentation task.
  • Frozen Knowledge: During the GAN training, the weights of the segmentation network (DI2I) are frozen. This prevents the segmenter from "cheating" by adapting to noisy, poorly translated images; instead, it forces the generator to produce high-quality, "segmentable" translations.
  • Leveraging 3D for 2D: The core "aha" moment is realizing that 3D CT scans are a goldmine for 2D X-ray training data, provided you have a smart enough "translator" to handle the visual discrepancies.

Key Results

The results demonstrate that "vanilla" models fail in the real world, but task-driven adaptation nearly closes the gap:

Method Mean Dice Score (Accuracy)
Vanilla DI2I (Trained on DRR, tested on X-ray) 30.8% (Complete failure)
Standard CycleGAN 80.8%
TD-GAN (Proposed) 85.4%
Supervised Training (Trained directly on labeled X-rays) 88.3%

By using TD-GAN, the researchers achieved 85% accuracy without using any manual labels for the X-ray images, coming within 3% of the performance of a model trained on expensive, manually-labeled data.

Limitations and Future Work

  • Clinical Availability: As noted in the paper's disclaimer, this is a research feature and not yet commercially available or regulatory-cleared.
  • Scope of Organs: The study focused on four major organs. Extending this to smaller or more variable structures (like specific lesions or complex vascular structures) remains a challenge.
  • Generalization: While the framework is general, the authors suggest future work could adapt it to other tasks like anatomical landmark localization or detecting abnormal motion in fluoroscopic sequences.

Summary updated

Abstract

Automatic parsing of anatomical objects in X-ray images is critical to many clinical applications in particular towards image-guided invention and workflow automation. Existing deep network models require a large amount of labeled data. However, obtaining accurate pixel-wise labeling in X-ray images relies heavily on skilled clinicians due to the large overlaps of anatomy and the complex texture patterns. On the other hand, organs in 3D CT scans preserve clearer structures as well as sharper boundaries and thus can be easily delineated. In this paper, we propose a novel model framework for learning automatic X-ray image parsing from labeled CT scans. Specifically, a Dense Image-to-Image network (DI2I) for multi-organ segmentation is first trained on X-ray like Digitally Reconstructed Radiographs (DRRs) rendered from 3D CT volumes. Then we introduce a Task Driven Generative Adversarial Network (TD-GAN) architecture to achieve simultaneous style transfer and parsing for unseen real X-ray images. TD-GAN consists of a modified cycle-GAN substructure for pixel-to-pixel translation between DRRs and X-ray images and an added module leveraging the pre-trained DI2I to enforce segmentation consistency. The TD-GAN framework is general and can be easily adapted to other learning tasks. In the numerical experiments, we validate the proposed model on 815 DRRs and 153 topograms. While the vanilla DI2I without any adaptation fails completely on segmenting the topograms, the proposed model does not require any topogram labels and is able to provide a promising average dice of $$85\%$$ which achieves the same level accuracy of supervised training (88%).
BibTeX
@inproceedings{zhang2018task,
  author = {Yue Zhang and Shun Miao and Tommaso Mansi and Rui Liao},
  title = {{Task Driven Generative Modeling for Unsupervised Domain Adaptation: Application to X-ray Image Segmentation}},
  booktitle = {International Conference on Medical Image Computing and Computer-Assisted Intervention},
  year = {2018},
  url = {https://openreview.net/forum?id=-iwt4lT3PM},
}

Loading related content...

Figure