TL;DR
InPK (Infusing Prior Knowledge) improves how Vision-Language Models (VLMs) like CLIP adapt to new tasks by replacing randomly initialized prompt tokens with class-specific knowledge generated by LLMs. By hierarchically "infusing" this knowledge across multiple encoder layers and using a novel text-to-vision projection, the model achieves significantly better generalization on unseen classes and robustness across diverse datasets.
What Problem Does This Paper Address?
Vision-Language Models (VLMs) like CLIP are incredibly powerful, but adapting them to specific tasks (like identifying rare bird species or satellite imagery) usually requires "prompt tuning." Standard methods like CoOp replace fixed text prompts with "learnable tokens"—vectors that the model optimizes during training.
However, these learnable tokens usually start as random noise. This creates two major hurdles:
- Optimization Trajectory: Starting from random noise is like trying to find a specific house in a city without a map; the model often gets stuck in "suboptimal" solutions that don't represent the actual semantics of the class.
- Overfitting: Because these tokens are optimized only on a few training examples ("base classes"), they tend to overfit. When the model encounters a "novel" (unseen) class, the learned prompts fail to generalize, leading to poor performance on anything outside the narrow training set.
Methods
The authors propose InPK, which moves away from random initialization toward a knowledge-driven approach.
- Prior Knowledge Generation: Before training, the authors use GPT-4 to generate a list of discriminative attributes for every class in a dataset (e.g., for a "Persian cat," attributes might include "long fluffy fur" and "flat face").
- Hierarchical Infusion: Instead of just concatenating this knowledge at the input, InPK uses an attribute-aware attention mechanism. This "infuses" the LLM-generated knowledge into the learnable tokens. Crucially, this isn't a one-time injection; the authors reinforce this interaction across the first $J$ layers of the transformer to prevent the class information from being "diluted" as it passes through the deep network.
- Learnable Text-to-Vision Projection: To ensure the visual and textual branches stay aligned after these prompt adjustments, they introduce a learnable projection layer ($F$) that maps textual tokens into the visual prompt space.
- Class-Name Saliency Regularization: To prevent the model from focusing too much on generic attributes (like "has four legs") and forgetting the unique identity of the class name, they apply an L1 loss constraint. This keeps the prompted features consistent with the original CLIP feature space.
Datasets
The model was evaluated on a rigorous suite of 11 recognition datasets covering diverse domains:
- General Objects: ImageNet, Caltech101.
- Fine-grained: OxfordPets, StanfordCars, Flowers102, Food101, FGVCAircraft.
- Specialized Tasks: SUN397 (scenes), UCF101 (actions), DTD (textures), and EuroSAT (satellite imagery).
Key Insights and Ideas
- Initialization Matters: The "Aha!" moment here is that random initialization is a liability. By constraining learnable tokens to a "semantically meaningful subspace" from the start, the model avoids irrelevant regions of the embedding space.
- Combating Information Dilution: In deep transformers, specific class information tends to wash out in later layers. By repeatedly re-injecting prior knowledge (progressive interaction), InPK maintains a "semantic anchor" throughout the encoding process.
- Bridging the Gap to Unseen Classes: For novel classes, the model uses shared visual concepts (attributes) as a bridge. Even if the model hasn't seen a "Tiger," it understands "stripes" and "feline features" from its prior knowledge, allowing it to infer the correct position in the semantic space.
Key Results
InPK sets a new state-of-the-art (SOTA) across several benchmarks:
- Base-to-Novel Generalization: On average across 11 datasets, InPK achieved a Harmonic Mean (HM) of 81.85%, outperforming the previous SOTA (TCP) by 2.34%.
- Novel Class Boost: The most significant gain was on unseen classes, where accuracy improved by 3.43% on average. On the DTD (texture) dataset, the jump was a massive 13.23%.
- Few-Shot Efficiency: InPK consistently outperformed competitors in 1, 2, 4, 8, and 16-shot settings, proving that prior knowledge effectively compensates for a lack of labeled images.
- Cross-Dataset Robustness: When trained on ImageNet and tested on 10 other datasets, InPK achieved an average accuracy of 69.90%, showing it doesn't just memorize one distribution.
Limitations and Future Work
- Static Knowledge: The prior knowledge is generated offline by an LLM. If the LLM has a "hallucination" or provides biased attributes for a rare class, the VLM's performance might suffer.
- Prompt Depth Sensitivity: The authors found that infusing knowledge beyond the 9th layer actually decreased performance, suggesting that there is a limit to how much "guidance" the high-level features need before they become over-constrained.
- Future Direction: Integrating a real-time, dynamic knowledge retrieval system (instead of offline generation) could allow the model to adapt to even more obscure or evolving categories without manual intervention.
Summary updated