Abstract
Foundation models pretrained on large-scale 3D medical imaging data face challenges when adapted to multiple downstream tasks under continual learning with limited labeled data. We address few-shot continual learning for 3D brain MRI by combining a frozen pretrained backbone with task-specific Low-Rank Adaptation (LoRA) modules. Tasks arrive sequentially -- tumor segmentation (BraTS) and brain age estimation (IXI) -- with no replay of previous task data. Each task receives a dedicated LoRA adapter; only the adapter and task-specific head are trained while the backbone remains frozen, thereby eliminating catastrophic forgetting by design (BWT=0). In continual learning, sequential full fine-tuning suffers severe forgetting (T1 Dice drops from 0.80 to 0.16 after T2), while sequential linear probing achieves strong T1 (Dice 0.79) but fails on T2 (MAE 1.45). Our LoRA approach achieves the best balanced performance across both tasks: T1 Dice 0.62$\pm$0.07, T2 MAE 0.16$\pm$0.05, with zero forgetting and <0.1% trainable parameters per task, though with noted systematic age underestimation in T2 (Wilcoxon $p<0.001$). Frozen foundation models with task-specific LoRA adapters thus offer a practical solution when both tasks must be maintained under few-shot continual learning.
Introduction
In clinical settings, AI models must often adapt to new tasks over time. For instance, a hospital might deploy a model for tumor segmentation and later require the addition of brain age estimation. However, standard deep learning models typically suffer from catastrophic forgetting: training on a new task (Task B) often overwrites the parameters necessary for a previous task (Task A).
This challenge is intensified in medical imaging by data scarcity and privacy constraints. Labeled medical data is expensive and rare, leading to "few-shot" learning scenarios. Furthermore, HIPAA regulations and storage limits often prevent the "replay" of old data to mitigate forgetting.
Methodology: Modular Adaptation with LoRA
To address these issues, we propose a framework that combines a frozen foundation model with Low-Rank Adaptation (LoRA). We utilize a FOMO-style 3D UNet backbone, which remains completely frozen throughout the learning process.
For each sequential task, we inject dedicated LoRA modules into the network's layers. Specifically, we use 1x1x1 convolutions as LoRA modules to maintain compatibility with the 3D architecture. Because each task has its own isolated adapter and the backbone is never modified, performance on Task 1 is mathematically guaranteed to remain constant after training on Task 2. This results in a Backward Transfer (BWT) of exactly zero, eliminating forgetting by design.
Experimental Setup
The approach was evaluated on two distinct 3D brain MRI tasks:
- Task 1 (Segmentation): Identifying tumor boundaries using the BraTS 2023 Glioma dataset (1,251 cases).
- Task 2 (Regression): Estimating brain age using the IXI dataset (~577 subjects).
Experiments were conducted in a "few-shot" regime, using only 16, 32, or 64 labeled samples per task during adaptation.
Results and Discussion
The LoRA approach provided the most balanced performance across tasks without forgetting. While sequential full fine-tuning caused Task 1 Dice scores to collapse from 0.80 to 0.16 after learning Task 2, LoRA maintained a stable performance (0.62 Dice score with 32 samples).
Key Findings
- Efficiency: Each new task requires training fewer than 50,000 parameters (<0.1% of the 50M parameter backbone), making the system lightweight and fast to adapt.
- Depth of Adaptation: Linear probing (training only the final head) performed well for segmentation but failed for regression (MAE 1.45). LoRA provided the necessary depth to handle both task types effectively (MAE 0.16).
- Robustness: Standard continual learning methods like EWC or Replay often overfitted to the small few-shot validation sets in regression tasks, whereas LoRA remained more robust.
Limitations and Future Work
While successful in preventing forgetting, several challenges remain. A performance gap exists between LoRA (0.62 Dice) and full fine-tuning (0.80), highlighting a trade-off between memory and peak performance. Additionally, the model exhibited a systematic bias in underestimating brain age, likely due to the few-shot regime and demographic specifics of the IXI dataset. Future work will explore the impact of task order and the inclusion of more diverse benchmarks.
Summary updated