TL;DR
This paper introduces ECSA, a novel framework for Medical Visual Question Answering (Med-VQA) that tackles two major challenges: poor performance on rare diseases (few-shot generalization) and forgetting previously learned information when acquiring new knowledge (catastrophic forgetting). ECSA achieves state-of-the-art results by using specialized modules to better distinguish visually similar but clinically different cases and to store knowledge without retraining.
What Problem Does This Paper Address?
Medical Visual Question Answering (Med-VQA) systems aim to help doctors by answering questions about medical images. However, developing these systems faces two significant hurdles. First, getting enough high-quality, labeled medical data is incredibly difficult and expensive, leading to a scarcity of data for rare diseases. This makes it hard for models to generalize well to these "few-shot" scenarios. Second, medical knowledge is constantly evolving. AI models need to learn new information continuously without forgetting what they already know – a problem known as "catastrophic forgetting." Existing solutions often tackle these issues separately, but a unified approach is needed for robust clinical decision support.
Methods
ECSA is built upon powerful pre-trained vision (BiomedCLIP) and language (Flan-T5) models. Its novelty lies in two core modules:
- Clinical-Semantic Disambiguation Module (CSDM): This module uses a clever "debiased hard negative mining" strategy. In contrastive learning, models learn by distinguishing similar and dissimilar examples. "Hard negatives" are examples that look similar but are clinically different. CSDM specifically identifies and correctly handles these tricky cases, improving the model's ability to understand subtle differences in medical images, which is crucial for few-shot learning.
- Prompt-based Knowledge Consolidation Module (PKC): This acts as a "rehearsal-free" memory. Instead of retraining the entire model on new data (which causes forgetting), PKC stores learned knowledge as "soft prompts" in a dynamic, non-parametric memory. This allows the model to access and use past knowledge without overwriting it, effectively preventing catastrophic forgetting.
Datasets
The following datasets were used for training and evaluation:
- VQA-RAD: Radiology VQA dataset.
- SLAKE: A larger, bilingual Med-VQA dataset.
- PathVQA: Pathology image VQA dataset.
- VQA-Med-2019: Radiology images dataset.
- ROCO-v2: Used as an external knowledge base for the PKC module.
Key Insights and Ideas
- The "False Negative Trap" is a critical issue in medical contrastive learning: Visually similar but clinically distinct cases can mislead models, causing them to "forget" crucial diagnostic features. ECSA's debiased hard negative mining directly addresses this.
- A unified framework is more effective than isolated solutions: ECSA demonstrates that tackling few-shot generalization and continual learning simultaneously leads to superior performance and robustness.
- Parameter-efficient learning is key for large foundation models: By freezing most of the pre-trained backbone and only training smaller modules, ECSA achieves high performance with significantly fewer trainable parameters.
- Non-parametric memory stores are powerful for continual learning: The PKC module's ability to store knowledge externally without retraining is crucial for preventing catastrophic forgetting.
Key Results
ECSA achieved state-of-the-art or highly competitive performance across four benchmark datasets:
- VQA-RAD: 80.15% overall accuracy.
- SLAKE: 85.10% overall accuracy.
- PathVQA: 64.57% overall accuracy (demonstrating generalization to a new domain).
- VQA-Med-2019: 82.23% overall accuracy.
In a challenging continual learning scenario (five tasks), ECSA achieved a remarkably low forgetting rate of 13.50%, significantly outperforming baselines. Ablation studies confirmed that the PKC module was primarily responsible for this knowledge retention.
Limitations and Future Work
- Scalability and Generalizability: The framework was tested on standard benchmarks. Its performance on larger, institution-specific datasets and a wider range of imaging modalities needs further validation.
- Integrating Structured Knowledge: Future work could explore incorporating medical knowledge graphs into the PKC module to enhance complex reasoning capabilities.
- Real-world Clinical Validation: The ultimate value of ECSA in real-world clinical workflows needs to be verified through large-scale clinical validation.
Summary updated