Alternating Training-based Label Smoothing Enhances Prompt Generalization

Yang Chen, Yanbin Wei, Ke Jin, Yi Kong, James T. Kwok, Yu ZhangarXiv 2025 · DOI · arXiv

Introduction

Vision-Language Models (VLMs) like CLIP have demonstrated remarkable zero-shot capabilities. However, adapting these models to specific downstream tasks often requires fine-tuning. Prompt tuning (e.g., CoOp) has emerged as a parameter-efficient alternative, learning a small set of "prompt vectors" while keeping the model's core parameters frozen.

A significant challenge in prompt tuning is overfitting: learned prompts often lose the model's inherent ability to generalize to unseen classes or domains. While Label Smoothing (LS) is a standard regularization technique, empirical evidence suggests that "vanilla" LS—replacing hard one-hot labels with uniform soft distributions—actually impairs prompt tuning. This performance drop likely stems from a distribution shift; since pre-trained VLMs are typically trained with hard labels, switching entirely to soft labels during fine-tuning weakens the learned prompts' alignment with the model's foundational knowledge.

Methodology

To address these issues, the authors introduce ATLaS (Alternating Training-based Label Smoothing). This approach combines a novel training schedule with semantically rich soft labels.

Alternating Training Schedule

Instead of constant smoothing, ATLaS alternates between hard and soft labels. For $K-1$ epochs, the model is trained using standard one-hot labels to maintain the sharp discriminative features of the original VLM. In every $K$-th epoch, the model switches to soft labels for regularization. This strategy ensures the prompts remain grounded in the pre-trained knowledge while benefiting from the smoothing effect.

Offline Soft Label Generation

Rather than using uniform distributions, ATLaS leverages CLIP’s multimodal knowledge to generate "informed" soft labels offline:

  • Class-wise Soft Labels (CSL): These utilize textual similarities between class embeddings. For example, a "husky" label will assign higher probability to "wolf" than to "car."
  • Instance-wise Soft Labels (ISL): These are image-specific, using CLIP’s zero-shot predictions to capture intra-class variance, such as a dog that visually resembles a fox.
Overview of the ATLaS framework, illustrating the alternating supervision and the generation of class-wise and instance-wise soft labels.

Experimental Results

The researchers evaluated ATLaS across 11 image classification datasets, including ImageNet, StanfordCars, and Food101, as well as various ImageNet distribution shift variants.

Generalization Performance

ATLaS consistently outperforms baseline prompt tuning methods. In base-to-new class generalization, CoOp + ATLaS-ISL improved new-class accuracy from 63.23% to 73.62%, a significant ~10% gain. Furthermore, training on ImageNet and transferring to 10 other datasets yielded an average accuracy improvement of 2.10%.

Test accuracy over training epochs. ATLaS maintains high generalization performance compared to standard prompt tuning and other label smoothing variants.

Training Dynamics

As shown in the training curves, standard methods often experience a decay in test accuracy as training progresses, indicating overfitting. ATLaS, through its alternating mechanism, maintains a higher and more stable test accuracy throughout the training process. The oscillations in the ATLaS curve correspond to the periodic application of soft labels, which effectively regularizes the prompt learning.

Discussion and Conclusion

The success of ATLaS highlights the Consistency Principle: fine-tuning should remain consistent with the pre-training objective. By alternating back to hard labels and using multimodal supervision, ATLaS prevents prompts from drifting away from the VLM's semantic core.

While ATLaS shows broad compatibility with various prompt tuning methods (CoCoOp, MaPLe, etc.), it exhibits diminishing returns when paired with methods like PromptSRC that already incorporate similar multimodal insights. Future research may explore the application of ATLaS to larger VLMs and non-classification tasks such as object detection. Overall, ATLaS provides a computationally efficient (adding <0.15% to training time) and effective way to enhance the robustness of learned prompts.

Summary updated

Abstract

Recent advances in pre-trained vision-language models have demonstrated remarkable zero-shot generalization capabilities. To further enhance these models' adaptability to various downstream tasks, prompt tuning has emerged as a parameter-efficient fine-tuning method. However, despite its efficiency, the generalization ability of prompt remains limited. In contrast, label smoothing (LS) has been widely recognized as an effective regularization technique that prevents models from becoming over-confident and improves their generalization. This inspires us to explore the integration of LS with prompt tuning. However, we have observed that the vanilla LS even weakens the generalization ability of prompt tuning. To address this issue, we propose the Alternating Training-based Label Smoothing (ATLaS) method, which alternately trains with standard one-hot labels and soft labels generated by LS to supervise the prompt tuning. Moreover, we introduce two types of efficient offline soft labels, including Class-wise Soft Labels (CSL) and Instance-wise Soft Labels (ISL), to provide inter-class or instance-class relationships for prompt tuning. The theoretical properties of the proposed ATLaS method are analyzed. Extensive experiments demonstrate that the proposed ATLaS method, combined with CSL and ISL, consistently enhances the generalization performance of prompt tuning. Moreover, the proposed ATLaS method exhibits high compatibility with prevalent prompt tuning methods, enabling seamless integration into existing methods.
BibTeX
@misc{chen2025alternating,
  author = {Yang Chen and Yanbin Wei and Ke Jin and Yi Kong and James T. Kwok and Yu Zhang},
  title = {{Alternating Training-based Label Smoothing Enhances Prompt Generalization}},
  year = {2025},
  eprint = {2508.17846},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2508.17846},
}
Figure