ECSA: Mitigating Catastrophic Forgetting and Few-Shot Generalization in Medical Visual Question Answering

Qiaodong Jia, Shuxian Liu, Mingliang Chen, Tianyi Li, Jing YangTomography 2025 · DOI

TL;DR

This paper introduces ECSA, a novel framework for Medical Visual Question Answering (Med-VQA) that tackles two major challenges: poor performance on rare diseases (few-shot generalization) and forgetting previously learned information when acquiring new knowledge (catastrophic forgetting). ECSA achieves state-of-the-art results by using specialized modules to better distinguish visually similar but clinically different cases and to store knowledge without retraining.

What Problem Does This Paper Address?

Medical Visual Question Answering (Med-VQA) systems aim to help doctors by answering questions about medical images. However, developing these systems faces two significant hurdles. First, getting enough high-quality, labeled medical data is incredibly difficult and expensive, leading to a scarcity of data for rare diseases. This makes it hard for models to generalize well to these "few-shot" scenarios. Second, medical knowledge is constantly evolving. AI models need to learn new information continuously without forgetting what they already know – a problem known as "catastrophic forgetting." Existing solutions often tackle these issues separately, but a unified approach is needed for robust clinical decision support.

Methods

ECSA is built upon powerful pre-trained vision (BiomedCLIP) and language (Flan-T5) models. Its novelty lies in two core modules:

  • Clinical-Semantic Disambiguation Module (CSDM): This module uses a clever "debiased hard negative mining" strategy. In contrastive learning, models learn by distinguishing similar and dissimilar examples. "Hard negatives" are examples that look similar but are clinically different. CSDM specifically identifies and correctly handles these tricky cases, improving the model's ability to understand subtle differences in medical images, which is crucial for few-shot learning.
  • Prompt-based Knowledge Consolidation Module (PKC): This acts as a "rehearsal-free" memory. Instead of retraining the entire model on new data (which causes forgetting), PKC stores learned knowledge as "soft prompts" in a dynamic, non-parametric memory. This allows the model to access and use past knowledge without overwriting it, effectively preventing catastrophic forgetting.

Datasets

The following datasets were used for training and evaluation:

  • VQA-RAD: Radiology VQA dataset.
  • SLAKE: A larger, bilingual Med-VQA dataset.
  • PathVQA: Pathology image VQA dataset.
  • VQA-Med-2019: Radiology images dataset.
  • ROCO-v2: Used as an external knowledge base for the PKC module.

Key Insights and Ideas

  • The "False Negative Trap" is a critical issue in medical contrastive learning: Visually similar but clinically distinct cases can mislead models, causing them to "forget" crucial diagnostic features. ECSA's debiased hard negative mining directly addresses this.
  • A unified framework is more effective than isolated solutions: ECSA demonstrates that tackling few-shot generalization and continual learning simultaneously leads to superior performance and robustness.
  • Parameter-efficient learning is key for large foundation models: By freezing most of the pre-trained backbone and only training smaller modules, ECSA achieves high performance with significantly fewer trainable parameters.
  • Non-parametric memory stores are powerful for continual learning: The PKC module's ability to store knowledge externally without retraining is crucial for preventing catastrophic forgetting.

Key Results

ECSA achieved state-of-the-art or highly competitive performance across four benchmark datasets:

  • VQA-RAD: 80.15% overall accuracy.
  • SLAKE: 85.10% overall accuracy.
  • PathVQA: 64.57% overall accuracy (demonstrating generalization to a new domain).
  • VQA-Med-2019: 82.23% overall accuracy.

In a challenging continual learning scenario (five tasks), ECSA achieved a remarkably low forgetting rate of 13.50%, significantly outperforming baselines. Ablation studies confirmed that the PKC module was primarily responsible for this knowledge retention.

Limitations and Future Work

  • Scalability and Generalizability: The framework was tested on standard benchmarks. Its performance on larger, institution-specific datasets and a wider range of imaging modalities needs further validation.
  • Integrating Structured Knowledge: Future work could explore incorporating medical knowledge graphs into the PKC module to enhance complex reasoning capabilities.
  • Real-world Clinical Validation: The ultimate value of ECSA in real-world clinical workflows needs to be verified through large-scale clinical validation.

Summary updated

Abstract

<b>Objective:</b> Medical Visual Question Answering (Med-VQA), a key technology that integrates computer vision and natural language processing to assist in clinical diagnosis, possesses significant potential for enhancing diagnostic efficiency and accuracy. However, its development is constrained by two major bottlenecks: weak few-shot generalization capability stemming from the scarcity of high-quality annotated data and the problem of catastrophic forgetting when continually learning new knowledge. Existing research has largely addressed these two challenges in isolation, lacking a unified framework. <b>Methods:</b> To bridge this gap, this paper proposes a novel Evolvable Clinical-Semantic Alignment (ECSA) framework, designed to synergistically solve these two challenges within a single architecture. ECSA is built upon powerful pre-trained vision (BiomedCLIP) and language (Flan-T5) models, with two innovative modules at its core. First, we design a Clinical-Semantic Disambiguation Module (CSDM), which employs a novel debiased hard negative mining strategy for contrastive learning. This enables the precise discrimination of "hard negatives" that are visually similar but clinically distinct, thereby significantly enhancing the model's representation ability in few-shot and long-tail scenarios. Second, we introduce a Prompt-based Knowledge Consolidation Module (PKC), which acts as a rehearsal-free non-parametric knowledge store. It consolidates historical knowledge by dynamically accumulating and retrieving task-specific "soft prompts," thus effectively circumventing catastrophic forgetting without relying on past data. <b>Results:</b> Extensive experimental results on four public benchmark datasets, VQA-RAD, SLAKE, PathVQA, and VQA-Med-2019, demonstrate ECSA's state-of-the-art or highly competitive performance. Specifically, ECSA achieves excellent overall accuracies of 80.15% on VQA-RAD and 85.10% on SLAKE, while also showing strong generalization with 64.57% on PathVQA and 82.23% on VQA-Med-2019. More critically, in continual learning scenarios, the framework achieves a low forgetting rate of just 13.50%, showcasing its significant advantages in knowledge retention. <b>Conclusions:</b> These findings validate the framework's substantial potential for building robust and evolvable clinical decision support systems.
BibTeX
@inproceedings{jia2025ecsa,
  author = {Qiaodong Jia and Shuxian Liu and Mingliang Chen and Tianyi Li and Jing Yang},
  title = {{ECSA: Mitigating Catastrophic Forgetting and Few-Shot Generalization in Medical Visual Question Answering}},
  booktitle = {Tomography},
  year = {2025},
  doi = {10.3390/tomography11100115},
}
Figure