Introduction
In the rapidly evolving landscape of artificial intelligence, a primary goal is to develop models that emulate human-like learning—continuously acquiring new skills without succumbing to "catastrophic forgetting." While Continual Learning (CL) has seen significant progress in uni-modal tasks like image classification, applying these principles to Visual Question Answering (VQA) presents unique challenges. VQA requires sophisticated reasoning across two distinct modalities: vision and language.
Researchers from Tsinghua University and Alibaba Group have introduced TRIPLET, a framework designed to address the complexities of multi-modal continual learning. TRIPLET enables VQA models to learn a sequence of new tasks without requiring a rehearsal buffer (storing old data), achieving state-of-the-art performance with minimal forgetting across diverse scenarios.
The Challenge of Multi-Modal Continual Learning
Most existing CL research focuses on uni-modal scenarios. When applied to VQA, these methods often fall short because they fail to account for the joint distribution of images and questions. The authors identify two primary limitations in current approaches:
- Partial Formulation: Existing benchmarks often vary only one modality (e.g., new images with old question types), failing to test the model's true flexibility in dynamic multi-modal environments.
- Neglected Interactions: VQA relies heavily on the interaction between modalities. Simply tuning one encoder while freezing another fails to capture the complex fusion necessary for accurate reasoning.
The TRIPLET Framework
TRIPLET (MulTi-Modal PRompt LearnIng with DecouPLing bEfore InTeraction) addresses these issues by using prompt learning on frozen, pre-trained backbones (such as ALBEF or FLAVA). This approach avoids fine-tuning the entire model, which is a primary cause of forgetting.
The framework operates in three core stages:
1. Decoupled Prompts
TRIPLET breaks down prompts into specific components to maintain clarity in representation:
- Modality-wise Decoupling: Separate prompts are maintained for Vision ($P^{(v)}$), Language ($P^{(q)}$), and Fusion ($P^{(f)}$).
- Complementary Prompts: Each prompt is split into a General Prompt for task-invariant knowledge and an Expert Prompt for task-specific nuances.
- Selective Deep Decoupling: Prompts are strategically attached to specific Transformer layers to optimize memory efficiency.
2. Prompt Interaction Strategies
To facilitate effective multi-modal reasoning, TRIPLET employs two interaction strategies:
- Modality-Interaction: Uses low-rank matrices to bridge different modality prompts, encouraging mutual information propagation.
- Task-Interaction: A constraint that stabilizes the interaction structure across tasks, preventing the overwriting of previously acquired knowledge.
3. Query-and-Match Inference
During inference, TRIPLET uses a "Query-and-Match" strategy. It compares input features against learned "keys" to automatically select the most relevant Expert Prompts for a given image-question pair, allowing the model to handle tasks without explicit task IDs.
Experimental Evaluation
To rigorously test TRIPLET, the authors established two new benchmarks: CL-TDIUC and CL-VQA2.0. These benchmarks cover three distinct scenarios:
- Continual Vision (ConVS): New visual scenes with existing question types.
- Continual Language (ConLS): Existing scenes with new types of questions.
- Continual Vision-Language (ConVLS): Simultaneous changes in both images and questions.
Performance and Results
TRIPLET consistently outperforms state-of-the-art methods like L2P and DualPrompt across all scenarios.
Key findings include:
- Superior Accuracy: On the challenging CL-VQA2.0 (ConVLS) scenario, TRIPLET achieved an average accuracy of 60.53%, significantly outperforming S-Prompts (46.36%).
- Near-Zero Forgetting: TRIPLET demonstrates remarkable stability. For instance, on CL-TDIUC (ConVS), its forgetting rate was only 0.15%, compared to 3.93% for DualPrompt.
- Efficiency: The framework is highly efficient, requiring only approximately 0.55% additional memory relative to the base model.
Conclusion
TRIPLET represents a significant advancement in multi-modal continual learning. By decoupling prompts before explicitly modeling their interactions, it provides a robust, exemplar-free solution for dynamic VQA environments. This "decouple and interact" philosophy offers a promising path forward for more complex multi-modal systems, such as video question answering and interactive dialogue agents.
Summary updated