Learning From Partially Labeled Data for Multi-Organ and Tumor Segmentation

Yutong Xie, Jianpeng Zhang, Yong Xia, Chunhua Shen

Medical image segmentation is a cornerstone of computer-aided diagnosis, but it faces a massive bottleneck: data labeling. It is incredibly labor-intensive for a radiologist to contour every organ and tumor in a 3D CT scan. Consequently, most public datasets are "partially labeled"—one might focus only on the liver, while another focuses only on the kidneys.

In this paper, Xie et al. present TransDoDNet, a Transformer-based architecture designed to learn from these fragmented datasets to create a unified, high-performance segmentation model.

TL;DR

TransDoDNet is a "dynamic on-demand" network that learns to segment multiple organs and tumors by training on several partially labeled datasets simultaneously. By using a Transformer to generate task-specific segmentation kernels on the fly, it captures global organ-wise dependencies and outperforms both traditional multi-head networks and its predecessor, DoDNet.

What Problem Does This Paper Address?

The "partial labeling issue" is a major hurdle in medical AI. If you try to train a single model on a Liver dataset and a Kidney dataset, a conflict arises: the Liver dataset treats kidneys as "background," while the Kidney dataset treats the liver as "background."

Standard solutions are often inefficient:

  1. One-net-per-task: Training separate models for every organ (computationally expensive and ignores anatomical relationships).
  2. Multi-head networks: A shared backbone with many "heads" (inflexible and scales poorly as you add more tasks).

The authors seek a single, unified architecture that can handle an arbitrary number of segmentation tasks without the computational bloat of multiple heads, all while leveraging the anatomical context that organs provide for one another.

Methods

The core innovation is the TransDoDNet (Transformer-based Dynamic On-Demand Network). Its architecture consists of:

  • Hybrid Backbone: A 3D CNN (based on a ResNet-50 style encoder-decoder) for local feature extraction, integrated with a Transformer at the bottleneck to model long-range, global spatial dependencies.
  • Dynamic Head: Instead of fixed weights, the final segmentation layer uses kernels that are generated on-demand.
  • Transformer-based Kernel Generator: This is the "brain" of the system. It takes "organ embeddings" (vectors representing specific organs) and processes them through a Transformer encoder-decoder. Because Transformers use self-attention, the generated kernel for the "Liver" task is "aware" of the features and dependencies of other organs like the "Spleen" or "Pancreas."
  • Deformable Attention: To handle the high computational cost of 3D volumes, the authors use multi-scale deformable attention (MSDA), which allows the model to focus on a small set of key sampling locations rather than every single voxel.

Datasets

The authors introduced a new large-scale benchmark called MOTS (Multi-Organ and Tumor Segmentation).

  • Composition: It aggregates seven partially labeled sub-datasets from sources like LiTS (Liver), KiTS (Kidney), and the Medical Segmentation Decathlon (MSD).
  • Scale: 1,155 3D abdominal CT scans (920 for training, 235 for testing).
  • Tasks: Covers 7 organs and their associated tumors (Liver, Kidney, Hepatic Vessel, Pancreas, Colon, Lung, and Spleen).
  • Downstream Evaluation: The model was also tested on BCV (Multi-Atlas Labeling) and BraTS (Brain Tumors) to prove its transfer learning capabilities.

Key Insights and Ideas

  • Organs are Neighbors: Traditional CNNs have a limited receptive field. By using a Transformer to generate kernels, the model learns that the position of the liver informs the expected position of the kidney. This "organ-wise dependency" is crucial for accurate segmentation in complex abdominal scans.
  • Dynamic vs. Static: By generating kernels on-the-fly based on a task ID, the model remains "single-head" in structure but "multi-task" in capability. This keeps the parameter count low while maintaining high flexibility.
  • Superiority over Self-Supervision: The authors found that pre-training on the partially labeled MOTS dataset provided a much stronger starting point for new tasks than BYOL (a popular self-supervised method). This suggests that even "partial" labels are significantly more valuable than no labels at all for medical tasks.

Key Results

  • State-of-the-Art Performance: TransDoDNet achieved a mean Dice score (mDice) of 78.26% on the MOTS benchmark, outperforming the original convolution-based DoDNet (76.94%) and multi-head approaches (76.33%).
  • Efficiency: Despite the added Transformer, the inference time remains low because the dynamic head is extremely lightweight. It is significantly faster than running seven individual networks.
  • Transfer Learning Boost: When transferred to the BCV dataset, the MOTS pre-trained TransDoDNet reached 83.19% mDice, beating models trained from scratch (81.53%) and those using BYOL pre-training (82.31%).
  • Robustness: Qualitative results (Fig. 9) show that TransDoDNet is particularly better at segmenting small tumors compared to its competitors.

Limitations and Future Work

  • Task Specificity: While the model handles multiple tasks, it still requires a "task condition" (telling the model which organ to look for). A fully "universal" model that segments everything in one pass without task prompts remains a future goal.
  • Computational Intensity: Although deformable attention helps, 3D Transformers are still memory-heavy during training, requiring careful optimization of patch sizes and resolutions.
  • Future Directions: The authors suggest extending this framework to even larger datasets and exploring "foundation models" for 3D medical imaging that can generalize across different modalities (CT, MRI, PET).

Summary updated

Abstract

Medical image benchmarks for the segmentation of organs and tumors suffer from the partially labeling issue due to its intensive cost of labor and expertise. Current mainstream approaches follow the practice of one network solving one task. With this pipeline, not only the performance is limited by the typically small dataset of a single task, but also the computation cost linearly increases with the number of tasks. To address this, we propose a Transformer based dynamic on-demand network (TransDoDNet) that learns to segment organs and tumors on multiple partially labeled datasets. Specifically, TransDoDNet has a hybrid backbone that is composed of the convolutional neural network and Transformer. A dynamic head enables the network to accomplish multiple segmentation tasks flexibly. Unlike existing approaches that fix kernels after training, the kernels in the dynamic head are generated adaptively by the Transformer, which employs the self-attention mechanism to model long-range organ-wise dependencies and decodes the organ embedding that can represent each organ. We create a large-scale partially labeled Multi-Organ and Tumor Segmentation benchmark, termed MOTS, and demonstrate the superior performance of our TransDoDNet over other competitors on seven organ and tumor segmentation tasks. This study also provides a general 3D medical image segmentation model, which has been pre-trained on the large-scale MOTS benchmark and has demonstrated advanced performance over current predominant self-supervised learning methods.
BibTeX
@article{xie2023learning,
  author = {Yutong Xie and Jianpeng Zhang and Yong Xia and Chunhua Shen},
  title = {{Learning From Partially Labeled Data for Multi-Organ and Tumor Segmentation}},
  journal = {IEEE Trans. Pattern Anal. Mach. Intell.},
  year = {2023},
  url = {https://openreview.net/forum?id=gsgsWErokC},
}

Loading related content...

Figure