Prompt Learning With Knowledge Regularization for Pre-Trained Vision-Language Models

Boyang Guo, Liang Li, Jiehua Zhang, Yaoqi Sun, Chenggang Yan, Xichun Sheng

Abstract

Prompt learning is an effective way to adapt pre-trained models to downstream tasks by training a small number of additional learnable prompts. Recent studies address several early challenges by combining generalized knowledge from frozen pre-trained VL models with task-specific knowledge from training data as guidance for prompt learning. However, existing methods still struggle with the generalization-adaptation (GA) trade-off dilemma: excessive reliance on generalized knowledge hinders adaptation to downstream tasks, while overemphasis on task-specific knowledge undermines the inherent generalization capabilities of pre-trained models. To address this issue, we propose a novel prompt learning method called Prompt Learning with Knowledge Regularization (PLKR). PLKR effectively mitigates the GA trade-off dilemma by offering greater flexibility in adapting to task-specific knowledge while minimizing the disruption of pre-trained knowledge. Specifically, we propose category-invariant and topology-invariant knowledge regularization to preserve generalized knowledge: the former enhances category-level discriminative capabilities while allowing flexible task-specific learning, and the latter maintains global topological stability during adaptation to new tasks. Through the proposed regularization, PLKR improves the performance on both base and new tasks. We evaluate the effectiveness of our approach on four representative tasks over 11 datasets. Experimental results show our method outperforms existing SOTA methods by a large margin.
BibTeX
@article{guo2026prompt,
  author = {Boyang Guo and Liang Li and Jiehua Zhang and Yaoqi Sun and Chenggang Yan and Xichun Sheng},
  title = {{Prompt Learning With Knowledge Regularization for Pre-Trained Vision-Language Models}},
  journal = {IEEE Trans. Multimedia},
  year = {2026},
  doi = {10.1109/tmm.2025.3639886},
}

Loading related content...

Figure