Decouple Before Interact: Multi-Modal Prompt Learning for Continual Visual Question Answering

Zi Qian, Xin Wang, Xuguang Duan, Pengda Qin, Yuhong Li, Wenwu Zhu

Introduction

In the rapidly evolving landscape of artificial intelligence, a primary goal is to develop models that emulate human-like learning—continuously acquiring new skills without succumbing to "catastrophic forgetting." While Continual Learning (CL) has seen significant progress in uni-modal tasks like image classification, applying these principles to Visual Question Answering (VQA) presents unique challenges. VQA requires sophisticated reasoning across two distinct modalities: vision and language.

Researchers from Tsinghua University and Alibaba Group have introduced TRIPLET, a framework designed to address the complexities of multi-modal continual learning. TRIPLET enables VQA models to learn a sequence of new tasks without requiring a rehearsal buffer (storing old data), achieving state-of-the-art performance with minimal forgetting across diverse scenarios.

The Challenge of Multi-Modal Continual Learning

Most existing CL research focuses on uni-modal scenarios. When applied to VQA, these methods often fall short because they fail to account for the joint distribution of images and questions. The authors identify two primary limitations in current approaches:

  1. Partial Formulation: Existing benchmarks often vary only one modality (e.g., new images with old question types), failing to test the model's true flexibility in dynamic multi-modal environments.
  2. Neglected Interactions: VQA relies heavily on the interaction between modalities. Simply tuning one encoder while freezing another fails to capture the complex fusion necessary for accurate reasoning.

The TRIPLET Framework

TRIPLET (MulTi-Modal PRompt LearnIng with DecouPLing bEfore InTeraction) addresses these issues by using prompt learning on frozen, pre-trained backbones (such as ALBEF or FLAVA). This approach avoids fine-tuning the entire model, which is a primary cause of forgetting.

The TRIPLET training and inference pipeline, illustrating decoupled prompts and interaction strategies.

The framework operates in three core stages:

1. Decoupled Prompts

TRIPLET breaks down prompts into specific components to maintain clarity in representation:

  • Modality-wise Decoupling: Separate prompts are maintained for Vision ($P^{(v)}$), Language ($P^{(q)}$), and Fusion ($P^{(f)}$).
  • Complementary Prompts: Each prompt is split into a General Prompt for task-invariant knowledge and an Expert Prompt for task-specific nuances.
  • Selective Deep Decoupling: Prompts are strategically attached to specific Transformer layers to optimize memory efficiency.

2. Prompt Interaction Strategies

To facilitate effective multi-modal reasoning, TRIPLET employs two interaction strategies:

  • Modality-Interaction: Uses low-rank matrices to bridge different modality prompts, encouraging mutual information propagation.
  • Task-Interaction: A constraint that stabilizes the interaction structure across tasks, preventing the overwriting of previously acquired knowledge.

3. Query-and-Match Inference

During inference, TRIPLET uses a "Query-and-Match" strategy. It compares input features against learned "keys" to automatically select the most relevant Expert Prompts for a given image-question pair, allowing the model to handle tasks without explicit task IDs.

Experimental Evaluation

To rigorously test TRIPLET, the authors established two new benchmarks: CL-TDIUC and CL-VQA2.0. These benchmarks cover three distinct scenarios:

  • Continual Vision (ConVS): New visual scenes with existing question types.
  • Continual Language (ConLS): Existing scenes with new types of questions.
  • Continual Vision-Language (ConVLS): Simultaneous changes in both images and questions.

Performance and Results

TRIPLET consistently outperforms state-of-the-art methods like L2P and DualPrompt across all scenarios.

Accuracy vs. Task ID for the TDIUC and VQA 2.0 benchmarks, demonstrating TRIPLET's superior performance.

Key findings include:

  • Superior Accuracy: On the challenging CL-VQA2.0 (ConVLS) scenario, TRIPLET achieved an average accuracy of 60.53%, significantly outperforming S-Prompts (46.36%).
  • Near-Zero Forgetting: TRIPLET demonstrates remarkable stability. For instance, on CL-TDIUC (ConVS), its forgetting rate was only 0.15%, compared to 3.93% for DualPrompt.
  • Efficiency: The framework is highly efficient, requiring only approximately 0.55% additional memory relative to the base model.

Conclusion

TRIPLET represents a significant advancement in multi-modal continual learning. By decoupling prompts before explicitly modeling their interactions, it provides a robust, exemplar-free solution for dynamic VQA environments. This "decouple and interact" philosophy offers a promising path forward for more complex multi-modal systems, such as video question answering and interactive dialogue agents.

Summary updated

Abstract

In the real world, a desirable Visual Question Answering model is expected to provide correct answers to new questions and images in a continual setting (recognized as CL-VQA). However, existing works formulate CL-VQA from a vision-only or language-only perspective, and straightforwardly apply the uni-modal continual learning (CL) strategies to this multi-modal task, which is improper and suboptimal. On the one hand, such a partial formulation may result in limited evaluations. On the other hand, neglecting the interactions between modalities will lead to poor performance. To tackle these challenging issues, we propose a comprehensive formulation for CL-VQA from the perspective of multi-modal vision-language fusion. Based on our formulation, we further propose MulTi-Modal PRompt LearnIng with DecouPLing bEfore InTeraction (TRIPLET), a novel approach that builds on a pre-trained vision-language model and consists of decoupled prompts and prompt interaction strategies to capture the complex interactions between modalities. In particular, decoupled prompts contain learnable parameters that are decoupled w.r.t different aspects, and the prompt interaction strategies are in charge of modeling interactions between inputs and prompts. Additionally, we build two CL-VQA benchmarks for a more comprehensive evaluation. Extensive experiments demonstrate that our TRIPLET outperforms state-of-the-art methods in both uni-modal and multi-modal continual settings for CL-VQA.
BibTeX
@inproceedings{qian2023decouple,
  author = {Zi Qian and Xin Wang and Xuguang Duan and Pengda Qin and Yuhong Li and Wenwu Zhu},
  title = {{Decouple Before Interact: Multi-Modal Prompt Learning for Continual Visual Question Answering}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2023},
  doi = {10.1109/iccv51070.2023.00276},
}
Figure