Task Driven Generative Modeling for Unsupervised Domain Adaptation: Application to X-ray Image Segmentation

Yue Zhang, Shun Miao, Tommaso Mansi, Rui Liao

Abstract

Automatic parsing of anatomical objects in X-ray images is critical to many clinical applications, particularly towards image-guided intervention and workflow automation. Existing deep network models require a large amount of labeled data. However, obtaining accurate pixel-wise labeling in X-ray images relies heavily on skilled clinicians due to the large overlaps of anatomy and the complex texture patterns. On the other hand, organs in 3D CT scans preserve clearer structures as well as sharper boundaries and thus can be easily delineated. In this paper, we propose a novel model framework for learning automatic X-ray image parsing from labeled CT scans. Specifically, a Dense Image-to-Image network (DI2I) for multi-organ segmentation is first trained on X-ray like Digitally Reconstructed Radiographs (DRRs) rendered from 3D CT volumes. Then we introduce a Task Driven Generative Adversarial Network (TD-GAN) architecture to achieve simultaneous style transfer and parsing for unseen real X-ray images. TD-GAN consists of a modified cycle-GAN substructure for pixel-to-pixel translation between DRRs and X-ray images and an added module leveraging the pre-trained DI2I to enforce segmentation consistency. The TD-GAN framework is general and can be easily adapted to other learning tasks. In the numerical experiments, we validate the proposed model on 815 DRRs and 153 topograms. While the vanilla DI2I without any adaptation fails completely on segmenting the topograms, the proposed model does not require any topogram labels and is able to provide a promising average dice of 85% which achieves the same level accuracy of supervised training (88%).

Introduction

Automatic segmentation (parsing) of organs in X-ray images is vital for guided surgery and diagnostic workflows. However, training these models presents a significant challenge. Manual pixel-wise labeling of X-rays is difficult because they are 2D projections where organs overlap, creating complex texture patterns that require expert clinical knowledge to delineate.

Conversely, 3D CT scans offer clear structures and sharp boundaries, making them a rich source of labeled data. The challenge lies in the "domain gap": models trained on synthetic X-rays derived from CT scans (Digitally Reconstructed Radiographs, or DRRs) typically fail when applied to real X-rays due to differences in noise, contrast, and texture. This paper addresses Unsupervised Domain Adaptation, proposing a method to transfer knowledge from the labeled synthetic domain to the unlabeled real X-ray domain.

Methodology

The proposed framework utilizes a two-stage pipeline to bridge the gap between simulation and reality.

1. The Segmenter (DI2I)

Initially, a Dense Image-to-Image network (DI2I) is trained. This encoder-decoder architecture uses "Dense Blocks" to improve feature extraction and gradient flow. It is trained to segment four key structures—lung, heart, liver, and bone—using only synthetic DRRs and their associated ground-truth labels derived from 3D CT volumes.

2. The Task Driven GAN (TD-GAN)

To adapt the DI2I for real X-rays, it is integrated into the TD-GAN architecture. This system performs simultaneous style transfer and parsing:

  • Style Transfer: Based on CycleGAN, it uses dual generators to translate images between the "DRR style" and "X-ray style."
  • Task-Driven Supervision: Unlike standard GANs that focus solely on visual realism, TD-GAN includes a task-driven discriminator. It evaluates whether a generated DRR (translated from a real X-ray) remains segmentable by the pre-trained, frozen DI2I network.
  • Consistency Losses: A "Cycle Segmentation Consistency" loss ensures that anatomical structures are preserved during the translation process.
Overview of the TD-GAN framework, illustrating the pipeline from 3D CT data generation to unsupervised domain adaptation for real X-ray images.

Results and Discussion

The model was validated using 815 synthetic DRRs and 153 real topograms (scout X-rays). Notably, the model never saw labels for the topograms during training. The results demonstrate that while a "vanilla" model fails in the real-world domain, the task-driven adaptation nearly closes the performance gap with supervised models.

Method Mean Dice Score (Accuracy)
Vanilla DI2I (No adaptation) 30.8%
Standard CycleGAN 80.8%
TD-GAN (Proposed) 85.4%
Supervised Training (Labeled X-rays) 88.3%

As shown in the qualitative results, the TD-GAN model successfully preserves anatomical edges and shapes, whereas the vanilla model produces noisy and inaccurate segmentations when faced with the domain shift of real X-ray data.

Qualitative segmentation results comparing the baseline (top row) with the proposed TD-GAN model (bottom row) across different anatomical structures.

Conclusion

TD-GAN provides a robust framework for unsupervised domain adaptation in medical imaging. By prioritizing anatomical consistency over simple aesthetic style transfer, the model achieves 85% accuracy without requiring manual X-ray labels. This approach effectively leverages existing 3D CT data to automate 2D X-ray parsing. Future work may extend this framework to more complex tasks, such as landmark localization or detecting abnormal motion in fluoroscopic sequences.

Summary updated

Abstract

Automatic parsing of anatomical objects in X-ray images is critical to many clinical applications in particular towards image-guided invention and workflow automation. Existing deep network models require a large amount of labeled data. However, obtaining accurate pixel-wise labeling in X-ray images relies heavily on skilled clinicians due to the large overlaps of anatomy and the complex texture patterns. On the other hand, organs in 3D CT scans preserve clearer structures as well as sharper boundaries and thus can be easily delineated. In this paper, we propose a novel model framework for learning automatic X-ray image parsing from labeled CT scans. Specifically, a Dense Image-to-Image network (DI2I) for multi-organ segmentation is first trained on X-ray like Digitally Reconstructed Radiographs (DRRs) rendered from 3D CT volumes. Then we introduce a Task Driven Generative Adversarial Network (TD-GAN) architecture to achieve simultaneous style transfer and parsing for unseen real X-ray images. TD-GAN consists of a modified cycle-GAN substructure for pixel-to-pixel translation between DRRs and X-ray images and an added module leveraging the pre-trained DI2I to enforce segmentation consistency. The TD-GAN framework is general and can be easily adapted to other learning tasks. In the numerical experiments, we validate the proposed model on 815 DRRs and 153 topograms. While the vanilla DI2I without any adaptation fails completely on segmenting the topograms, the proposed model does not require any topogram labels and is able to provide a promising average dice of 85% which achieves the same level accuracy of supervised training (88%).
BibTeX
@inproceedings{zhang2018task,
  author = {Yue Zhang and Shun Miao and Tommaso Mansi and Rui Liao},
  title = {{Task Driven Generative Modeling for Unsupervised Domain Adaptation: Application to X-ray Image Segmentation}},
  booktitle = {International Conference on Medical Image Computing and Computer-Assisted Intervention},
  year = {2018},
  url = {https://openreview.net/forum?id=-iwt4lT3PM},
}
Figure