An Empirical Analysis of Continual Learning for Heterogeneous Medical Visual Question Answering

Mai A. Shaaban, Tausifa Jan Saleem, Alaa Mohamed, Dilnaz Utemissova, Ufaq Khan, Mohammad YaqubarXiv 2026 · arXiv

TL;DR

Medical Visual Question Answering (MedVQA) systems must continuously adapt to new clinical tasks without forgetting old ones, but doing so across structurally diverse tasks remains a major challenge. This paper presents a systematic evaluation of continual learning (CL) methods for heterogeneous MedVQA, revealing that current parameter-level CL strategies struggle to maintain a stability-plasticity balance—especially when long-form text generation tasks disrupt the model's decoding behavior for short-answer tasks.

What Problem Does This Paper Address?

Deploying medical AI assistants in real-world clinical settings requires them to handle a wide variety of tasks, from diagnosing a tumor to counting cells or writing radiology reports. In practice, data for these tasks arrives incrementally over time. If you naively fine-tune a model on a new task, it suffers from catastrophic forgetting—it completely loses the ability to perform previously learned tasks.

While prior research in continual learning (CL) for medical imaging has looked at domain-incremental settings (e.g., learning the same classification task across different hospitals or imaging modalities), real-world clinical workflows require models to handle structurally heterogeneous tasks. These tasks differ fundamentally in their objectives, supervision granularity, and output formats (e.g., predicting a single class label vs. generating bounding box coordinates or free-form text). This paper addresses the critical gap in understanding how CL methods behave under these highly diverse optimization dynamics.

Fig. 1: CL for heterogeneous MedVQA tasks. An illustration of the desired stability-plasticity behavior, where the model is progressively updated to incor- porate new tasks while preserving previously acquired knowledge.

Methods

The authors freeze a pre-trained medical vision-language model (MedGemma-4b-it) and apply Low-Rank Adaptation (LoRA) to all linear layers. By keeping the backbone frozen and only updating the LoRA parameters, the stability-plasticity trade-off is isolated entirely within the adapter space.

They adapt five representative CL strategies to operate exclusively on these LoRA parameters:

  1. Replay: Maintains a small, balanced buffer (100 samples per task) of historical data to interleave with new task data.
  2. Elastic Weight Consolidation (EWC): Uses a Fisher information matrix to penalize changes to LoRA weights that were important for previous tasks.
  3. Knowledge Distillation (KD): Uses an output-level KL-divergence loss to force the current model (student) to match the predictions of the previous task's model (teacher).
  4. Soft-masking of Parameter-level Gradient flow (SPG): Softly dampens gradient updates on parameters critical to prior tasks based on gradient sensitivity.
  5. Orthogonal LoRA (O-LoRA): Allocates a distinct low-rank adapter per task and penalizes subspace overlap to enforce geometric orthogonality between tasks.

These are compared against two baselines: Multi-Task Learning (MTL, the upper bound where all data is available simultaneously) and Sequential Fine-Tuning (SeqFT, the lower bound with no forgetting mitigation).

Datasets

The experiments are conducted on the FLARE-MLLM-2D benchmark, which aggregates image-question-answer triplets from 19 public medical datasets. It spans eight imaging modalities (such as X-ray, Ultrasound, and Dermoscopy) and five distinct clinical task types:

  • Classification (evaluated via Balanced Accuracy)
  • Multi-label Classification (evaluated via Micro-F1)
  • Abnormality Detection (evaluated via F1 @ IoU 0.5)
  • Cell Counting (evaluated via Mean Absolute Error)
  • Report Generation (evaluated via the GREEN score for factual correctness)

Key Insights and Ideas

  • The "Report Generation" Curse: The authors discovered that task ordering heavily influences forgetting, primarily driven by long-form text generation. When a model learns "Report Generation" (an autoregressive, long-form task), it undergoes a massive decoding shift. Afterward, when prompted for short-answer tasks like classification or counting, it continues to generate long-form text, destroying its performance on those earlier tasks.
  • KD as a Decoding Anchor: Knowledge Distillation (KD) proved uniquely effective at mitigating this decoding shift because its output-level constraint directly prevents the model's text-generation style from drifting too far from the teacher model.
  • Parameter Constraints $\neq$ Functional Alignment: Simply constraining weight updates (as EWC, SPG, or O-LoRA do) is insufficient for heterogeneous tasks. Even if weights don't move much, the functional output behavior can change drastically when shifting between entirely different task structures.
  • Visualizing Weight Drift: By mapping the angular spread of LoRA parameter updates, the authors showed that Replay causes wide, unconstrained directional drift, whereas EWC and KD maintain tighter parameter distributions that align better with historical tasks.
Fig. 4: LoRA parameter directions across sequential tasks. Angles are computed in the high-dimensional LoRA space using Frobenius inner-product similarity between t −1 and t and visualized as 2D arrows via polar mapping. The dotted arrow shows the mean drift direction.

Key Results

  • SeqFT Fails Completely: Naive sequential fine-tuning results in near-total forgetting, with forgetting ratios (FR) approaching 1.0 (100% forgetting) on early tasks.
  • EWC is the Most Stable Overall: Across different task orderings (Easy $\rightarrow$ Hard, Hard $\rightarrow$ Easy, and Mixed), EWC demonstrated the most consistent stability-plasticity balance, maintaining reasonable accuracy on prior tasks without completely blocking the model's ability to learn new ones.
  • O-LoRA Struggles with Heterogeneity: Despite working well in standard NLP settings, O-LoRA suffered severe forgetting spikes here, showing extreme sensitivity to task ordering.
  • Detection is Universally Hard: Performance on the object detection task remained low across all CL methods and even the multi-task baseline, indicating that current medical VLMs struggle to output precise bounding box coordinates in a unified text format.

Limitations and Future Work

The authors note that existing parameter-level constraints are fundamentally limited when dealing with diverse task structures. Future robust MedVQA systems will require explicit functional alignment—methods designed to preserve output behavior and formatting constraints across arbitrary task sequences. Additionally, the study was limited to a 4B parameter model and 2D images; future work should explore how these dynamics scale to larger foundation models and complex 3D medical imaging tasks (like CT or MRI volumes).

Summary updated

Abstract

Deploying medical visual question answering (MedVQA) systems in real-world clinical settings requires models that adapt to new clinical tasks without forgetting previously acquired knowledge. Continual learning (CL) provides a practical framework for this setting. Despite rapid progress in medical vision-language models, the behavior of CL methods when training these models across heterogeneous MedVQA tasks remains underexplored. This work presents a systematic evaluation of CL for MedVQA across diverse clinical objectives, including classification, multi-label classification, detection, cell counting, and report generation. Specifically, we explore (1) the ability of existing CL methods to mitigate catastrophic forgetting; (2) their sensitivity to task ordering, analyzing how different task sequences influence performance retention and forgetting; and (3) the evolution of low-rank adaptation parameters as new tasks are learned, revealing patterns of weight drift under different CL methods. Our findings suggest that existing CL methods struggle to maintain stability-plasticity balance when tasks with different objectives and supervision formats are interleaved. Code and full experimental setup will be publicly available.
BibTeX
@misc{shaaban2026empirical,
  author = {Mai A. Shaaban and Tausifa Jan Saleem and Alaa Mohamed and Dilnaz Utemissova and Ufaq Khan and Mohammad Yaqub},
  title = {{An Empirical Analysis of Continual Learning for Heterogeneous Medical Visual Question Answering}},
  year = {2026},
  eprint = {2607.12048},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2607.12048},
}
Figure