Abstract
Medical image benchmarks for the segmentation of organs and tumors suffer from the partially labeling issue due to its intensive cost of labor and expertise. Current mainstream approaches follow the practice of one network solving one task. With this pipeline, not only the performance is limited by the typically small dataset of a single task, but also the computation cost linearly increases with the number of tasks. To address this, we propose a Transformer based dynamic on-demand network (TransDoDNet) that learns to segment organs and tumors on multiple partially labeled datasets. Specifically, TransDoDNet has a hybrid backbone that is composed of the convolutional neural network and Transformer. A dynamic head enables the network to accomplish multiple segmentation tasks flexibly. Unlike existing approaches that fix kernels after training, the kernels in the dynamic head are generated adaptively by the Transformer, which employs the self-attention mechanism to model long-range organ-wise dependencies and decodes the organ embedding that can represent each organ. We create a large-scale partially labeled Multi-Organ and Tumor Segmentation benchmark, termed MOTS, and demonstrate the superior performance of our TransDoDNet over other competitors on seven organ and tumor segmentation tasks. This study also provides a general 3D medical image segmentation model, which has been pre-trained on the large-scale MOTS benchmark and has demonstrated advanced performance over BYOL, the current predominant self-supervised learning method.
Introduction
Accurate segmentation of organs and tumors in 3D medical images is critical for clinical diagnosis and treatment planning. However, the field faces a significant "partial labeling" challenge. Due to the high cost and expertise required for annotation, most public datasets focus on a specific subset of organs (e.g., only the liver or only the kidneys). When training a single model on multiple such datasets, a conflict arises: an organ labeled in one dataset is treated as background in another.
The Challenge of Partial Labeling
Traditional approaches to multi-task segmentation typically fall into two categories:
- One-net-per-task: Training a separate model for each organ. This is computationally expensive and fails to capture anatomical relationships between organs.
- Multi-head networks: Using a shared backbone with separate output heads for each task. This approach is inflexible and scales poorly as the number of tasks increases.
Methodology: TransDoDNet
To overcome these limitations, we introduce TransDoDNet (Transformer-based Dynamic On-Demand Network). The architecture is designed to be a unified, flexible model capable of learning from fragmented datasets.
Hybrid Backbone and Dynamic Head
TransDoDNet utilizes a hybrid backbone combining a 3D CNN (ResNet-50 style) for local feature extraction with a Transformer at the bottleneck to capture long-range global spatial dependencies. The key innovation is the Dynamic Head, where the final segmentation kernels are not fixed weights but are generated on-demand based on the target task.
Transformer-based Kernel Generation
The "brain" of the system is a Transformer-based kernel generator. It takes "organ embeddings"—vectors representing specific organs—and processes them through a Transformer encoder-decoder. By utilizing self-attention, the generator ensures that the kernel for a specific task (e.g., "Liver") is informed by the features and spatial context of other organs (e.g., "Spleen" or "Pancreas"). To manage the computational load of 3D data, we employ multi-scale deformable attention (MSDA).
Experimental Results
We established the MOTS (Multi-Organ and Tumor Segmentation) benchmark, aggregating seven partially labeled datasets (including LiTS and KiTS) totaling 1,155 3D CT scans.
Performance and Efficiency
TransDoDNet achieved a mean Dice score (mDice) of 78.26% on the MOTS benchmark, significantly outperforming the original convolution-based DoDNet (76.94%) and standard multi-head approaches (76.33%). Despite the inclusion of a Transformer, the model remains efficient; the lightweight dynamic head allows for fast inference, far exceeding the speed of running multiple individual networks.
Transfer Learning
Pre-training on the MOTS benchmark proved highly effective for downstream tasks. On the BCV dataset, TransDoDNet reached 83.19% mDice, surpassing models trained from scratch or those using state-of-the-art self-supervised methods like BYOL. This highlights the immense value of leveraging partially labeled data for general medical image representation learning.
Conclusion
TransDoDNet provides a scalable and efficient solution for learning from partially labeled medical datasets. By integrating Transformers for both global feature modeling and dynamic kernel generation, the network captures vital anatomical dependencies that traditional CNNs miss. Future work will focus on developing fully universal models that can segment all organs in a single pass without explicit task conditioning.
Summary updated