Introduction
In clinical practice, medical AI systems are expected to evolve as new diagnostic needs arise. Ideally, a model should acquire new capabilities—such as identifying a newly relevant tumor type—without losing its proficiency in existing tasks, such as segmenting primary organs. However, deep learning models typically suffer from catastrophic forgetting, where training on new data leads to the overwriting of weights associated with previous knowledge.
While "rehearsal-based" methods (storing a subset of old data) can mitigate this, strict privacy regulations like HIPAA often prohibit the long-term storage or reuse of patient data across different training phases. This research introduces a novel framework designed to facilitate continual learning (CL) in medical image segmentation, allowing models to expand their repertoire of classes using only new data while maintaining high performance on previously learned tasks.
Methodology
The proposed framework addresses the dual challenges of knowledge retention and architectural scalability through two primary mechanisms: pseudo-labeling and image-aware, class-specific heads.
1. Pseudo-labeling for Knowledge Retention
To prevent the model from forgetting old classes when ground-truth labels for those classes are unavailable in new datasets, the researchers employ a pseudo-labeling strategy. When the model is presented with a new dataset (e.g., labeled only for "Kidneys"), the previous iteration of the model generates "soft" labels for the organs it already knows (e.g., "Liver" and "Spleen"). This allows the model to effectively re-learn old classes using the new images, maintaining a consistent internal representation without requiring access to the original training data.
2. CLIP-Driven Organ-Specific Heads
Rather than using a standard fixed output layer, the architecture utilizes a shared Swin UNETR backbone coupled with independent, lightweight segmentation heads for each class. This design features several innovations:
- Dynamic Parameter Generation: For every input image, a Global Average Pooling (GAP) layer extracts a feature vector. This vector is concatenated with a CLIP text embedding of the target organ's name (e.g., "a CT scan of a liver").
- MLP-Driven Kernels: A Multi-Layer Perceptron (MLP) processes this combined vector to generate specific convolution kernels for that organ's segmentation head. This ensures the segmentation process is conditioned on both the visual context and the semantic definition of the class.
- Decoupling and Efficiency: Because each organ has its own dedicated head, learning a new class causes minimal interference with the weights of existing heads. Furthermore, adding a new class is extremely efficient, requiring only approximately 0.12 GFLOPs of additional computation.
Experimental Setup
The framework was evaluated using three major abdominal CT datasets:
- BTCV: 47 images with 13 organ labels.
- LiTS: 130 images focused on liver and liver tumor segmentation.
- JHH: A large-scale private dataset from Johns Hopkins Hospital containing 200 scans with detailed annotations for organs, gastrointestinal tracts, and cardiovascular structures.
The researchers established "learning trajectories" where the model was trained sequentially: starting with standard organs (Step 1), moving to gastrointestinal tracts (Step 2), and concluding with cardiovascular structures or tumors (Step 3).
Results and Key Insights
The proposed method consistently outperformed established continual learning baselines, including LwF, ILT, and PLOP.
- Performance Stability: On the JHH dataset, the model maintained a Dice score of 0.787 for original organs at the end of the trajectory. In contrast, it achieved a Dice of 0.636 for the newest cardiovascular classes, significantly outperforming the best baseline (0.484).
- Tumor Segmentation: When transitioning from BTCV (organs) to LiTS (tumors), the model achieved a Dice of 0.466 for tumors while maintaining a high organ segmentation accuracy of 0.817.
- Semantic Guidance: Ablation studies confirmed that CLIP embeddings were critical. Replacing CLIP with simple one-hot encoding led to a performance drop, particularly in complex tasks like segmenting the gastrointestinal tract, suggesting that the semantic knowledge embedded in CLIP provides a superior initialization for new classes.
- Multi-label Flexibility: By using independent heads with Sigmoid activations rather than a global Softmax, the model can naturally handle overlapping labels, such as a tumor located within an organ.
Discussion and Future Work
While the framework demonstrates state-of-the-art performance, it is not without limitations. The reliance on pseudo-labels means that errors in early training stages can propagate through the learning trajectory. Additionally, while CLIP provides valuable semantic information, there remains a domain gap between general internet images and specialized medical structures.
Future research may explore extending this architecture toward open-vocabulary segmentation, potentially allowing models to segment organs they have never explicitly seen during training by leveraging text descriptions alone. This work represents a significant step toward more flexible, scalable, and privacy-compliant AI for medical imaging.
Summary updated