Abstract
Domain Incremental Learning (DIL) requires models to continuously adapt to a stream of new domains while preserving knowledge from previous ones. While prompt-based learning with frozen Vision Transformers (ViTs) has emerged as a leading strategy, existing methods often suffer from "component-wise misalignment." This occurs when prompts for different domains are learned independently, leading to structural inconsistencies that cause interference during prompt fusion. This paper introduces KA-Prompt (Componential Prompt-Knowledge Alignment), a framework that ensures prompt components are aligned across domains through intelligent initialization and structural consistency constraints. KA-Prompt significantly improves accuracy across multiple benchmarks without adding any computational overhead during inference.
Introduction
In the field of Domain Incremental Learning (DIL), an AI model must learn from a sequential stream of data where the underlying domain shifts—for example, transitioning from real-world photographs to sketches, then to paintings, and finally to clipart. The primary challenge is to adapt to new styles and environments without experiencing "catastrophic forgetting" of prior knowledge.
A popular and efficient approach to DIL involves prompt-based learning attached to a frozen Vision Transformer (ViT). Methods such as C-Prompt learn a unique set of prompts for each domain and then fuse these prompts during the inference stage to make predictions. However, this paper identifies a fundamental flaw in independent prompt learning: the lack of structural alignment between the components of different domain prompts.
The Problem: Component-wise Misalignment
The authors identify a phenomenon termed component-wise misalignment. Because prompts for each domain are typically learned in isolation, the internal structure of the prompt embeddings ends up randomly ordered. For instance, a specific part of a prompt embedding that learns to identify "geometric shapes" might be located at position 1 for Domain A, but at position 3 for Domain B.
When the model attempts to fuse these prompts during inference, it essentially mashes together mismatched structural components. This misalignment creates knowledge conflicts, limits the model's ability to share features across domains, and ultimately degrades predictive performance. The authors proved this bottleneck by demonstrating that randomly shuffling prompt components in baseline models leads to significant fluctuations in accuracy, confirming that default alignment is highly suboptimal.
The KA-Prompt Framework
To resolve this, the authors propose Componential Prompt-Knowledge Alignment (KA-Prompt). This method operates in two distinct phases during the training process:
1. Initial Componential Structure Configuring
Instead of initializing new domain prompts with random values, KA-Prompt performs Reusable Knowledge Mining. Using a greedy search algorithm, the system identifies historical prompts that contain shared knowledge relevant to the incoming domain. These historical prompts are used to seed the new prompts, establishing an intrinsic structural alignment from the very beginning of the learning process.
2. Online Alignment Preservation
Initial alignment can be lost as gradient updates cause prompt structures to drift during training. To prevent this, KA-Prompt employs an Aligning-guided New Prompt Learning scheme. As the model learns the new domain, it dynamically identifies the most similar historical prompts and applies a consistency constraint (an alignment loss). This acts as a structural anchor, allowing the model to learn domain-specific features while maintaining a compatible layout with previous knowledge.
Crucially, these alignment steps are restricted to the training phase. During inference, KA-Prompt uses the standard matching and fusion process, providing a performance boost with zero additional parameters or latency at deployment.
Experimental Evaluation
The authors evaluated KA-Prompt on four challenging multi-domain benchmarks:
- DomainNet: 345 classes across 6 style domains (Real, Quickdraw, Sketch, Painting, Infograph, Clipart).
- ImageNet-R: 200 categories across 15 different style domains.
- ImageNet-C: 200 categories featuring 15 types of quality corruptions and environmental changes.
- ImageNet-Mix: A massive 30-domain benchmark combining ImageNet-C and ImageNet-R to test extreme variations.
Results and Impact
KA-Prompt achieved state-of-the-art performance across all benchmarks, significantly outperforming the previous leading method, C-Prompt. Key results include:
- Accuracy Gains: An average accuracy boost of 4.73% across the four datasets, with specific gains of +5.59% on ImageNet-C and +5.00% on ImageNet-Mix.
- Efficiency: The method achieves these gains with zero additional learnable parameters compared to the baseline.
- Inference Performance: Because the alignment logic is contained within the training phase, there is no extra GPU memory or compute time required during inference, making it highly suitable for real-world applications.
Conclusion and Future Work
KA-Prompt demonstrates that the structural organization of prompts is just as important as the knowledge they contain. By enforcing componential alignment, the model can successfully combine generalizable knowledge across domains without interference.
While highly effective, the authors note that the Online Alignment Preservation module increases computational overhead during the training phase due to the constant calculation of similarities. Future research may focus on more efficient alignment loss calculations or extending this componential alignment philosophy to multi-modal models like CLIP, where aligning vision and text prompt components could further enhance cross-domain transfer.