Introduction
Vision-Language Models (VLMs) like CLIP have demonstrated remarkable zero-shot capabilities. However, adapting these models to specific downstream tasks often requires fine-tuning. Prompt tuning (e.g., CoOp) has emerged as a parameter-efficient alternative, learning a small set of "prompt vectors" while keeping the model's core parameters frozen.
A significant challenge in prompt tuning is overfitting: learned prompts often lose the model's inherent ability to generalize to unseen classes or domains. While Label Smoothing (LS) is a standard regularization technique, empirical evidence suggests that "vanilla" LS—replacing hard one-hot labels with uniform soft distributions—actually impairs prompt tuning. This performance drop likely stems from a distribution shift; since pre-trained VLMs are typically trained with hard labels, switching entirely to soft labels during fine-tuning weakens the learned prompts' alignment with the model's foundational knowledge.
Methodology
To address these issues, the authors introduce ATLaS (Alternating Training-based Label Smoothing). This approach combines a novel training schedule with semantically rich soft labels.
Alternating Training Schedule
Instead of constant smoothing, ATLaS alternates between hard and soft labels. For $K-1$ epochs, the model is trained using standard one-hot labels to maintain the sharp discriminative features of the original VLM. In every $K$-th epoch, the model switches to soft labels for regularization. This strategy ensures the prompts remain grounded in the pre-trained knowledge while benefiting from the smoothing effect.
Offline Soft Label Generation
Rather than using uniform distributions, ATLaS leverages CLIP’s multimodal knowledge to generate "informed" soft labels offline:
- Class-wise Soft Labels (CSL): These utilize textual similarities between class embeddings. For example, a "husky" label will assign higher probability to "wolf" than to "car."
- Instance-wise Soft Labels (ISL): These are image-specific, using CLIP’s zero-shot predictions to capture intra-class variance, such as a dog that visually resembles a fox.
Experimental Results
The researchers evaluated ATLaS across 11 image classification datasets, including ImageNet, StanfordCars, and Food101, as well as various ImageNet distribution shift variants.
Generalization Performance
ATLaS consistently outperforms baseline prompt tuning methods. In base-to-new class generalization, CoOp + ATLaS-ISL improved new-class accuracy from 63.23% to 73.62%, a significant ~10% gain. Furthermore, training on ImageNet and transferring to 10 other datasets yielded an average accuracy improvement of 2.10%.
Training Dynamics
As shown in the training curves, standard methods often experience a decay in test accuracy as training progresses, indicating overfitting. ATLaS, through its alternating mechanism, maintains a higher and more stable test accuracy throughout the training process. The oscillations in the ATLaS curve correspond to the periodic application of soft labels, which effectively regularizes the prompt learning.
Discussion and Conclusion
The success of ATLaS highlights the Consistency Principle: fine-tuning should remain consistent with the pre-training objective. By alternating back to hard labels and using multimodal supervision, ATLaS prevents prompts from drifting away from the VLM's semantic core.
While ATLaS shows broad compatibility with various prompt tuning methods (CoCoOp, MaPLe, etc.), it exhibits diminishing returns when paired with methods like PromptSRC that already incorporate similar multimodal insights. Future research may explore the application of ATLaS to larger VLMs and non-classification tasks such as object detection. Overall, ATLaS provides a computationally efficient (adding <0.15% to training time) and effective way to enhance the robustness of learned prompts.
Summary updated