Abstract
The emergence of vision-language foundation models, such as CLIP, has revolutionized image-text representation, enabling a broad range of applications via prompt learning. Despite its promise, real-world datasets often contain noisy labels that can degrade prompt learning performance. In this paper, we demonstrate that using mean absolute error (MAE) loss in prompt learning, named PromptMAE, significantly enhances robustness against noisy labels while maintaining high accuracy. Though MAE is straightforward and recognized for its robustness, it is rarely used in noisy-label learning due to its slow convergence and poor performance outside prompt learning scenarios. To elucidate the robustness of PromptMAE, we leverage feature learning theory to show that MAE can suppress the influence of noisy samples, thereby improving the signal-to-noise ratio and enhancing overall robustness. Additionally, we introduce PromptOT, a prompt-based optimal transport data purification method to enhance the robustness further. PromptOT employs text encoder representations in vision-language models as prototypes to construct an optimal transportation matrix. This matrix effectively partitions datasets into clean and noisy subsets, allowing for the application of cross-entropy loss to the clean subset and MAE loss to the noisy subset. Our Noise-Label Prompt Learning method, named NLPrompt, offers a simple and efficient approach that leverages the expressive representation and precise alignment capabilities of vision-language models for robust prompt learning. We validate NLPrompt through extensive experiments across various noise settings, demonstrating significant performance improvements.
Introduction
Vision-Language Models (VLMs) like CLIP have transformed computer vision by aligning visual and textual information in a shared latent space. To adapt these massive models to downstream tasks efficiently, researchers often employ prompt learning, which involves tuning a small set of continuous prompt vectors while keeping the backbone model frozen. However, this efficiency comes with a vulnerability: prompt learning is highly sensitive to noisy labels in the training data. In real-world settings, datasets are often imperfect, containing mislabeled samples that cause standard training objectives like Cross-Entropy (CE) loss to overfit to noise, degrading the model's generalization.
Methodology: NLPrompt
The proposed framework, NLPrompt, addresses this challenge through two primary components: PromptMAE and PromptOT.
PromptMAE: Robustness through MAE Loss
While Mean Absolute Error (MAE) loss is theoretically robust to noise, it is seldom used in deep learning because it typically suffers from slow convergence and poor performance when training entire networks. However, the authors discover that in the context of prompt learning—where only a few thousand parameters are updated—MAE converges rapidly and maintains high accuracy even under extreme noise. By suppressing the influence of noisy samples, PromptMAE improves the signal-to-noise ratio during optimization.
PromptOT: Data Purification via Optimal Transport
To further enhance performance, NLPrompt incorporates PromptOT, a data purification strategy. Unlike traditional methods that use random prototypes, PromptOT leverages the pre-trained text encoder of the VLM to generate class prototypes. It then uses Optimal Transport (OT) to compute a pseudo-label matrix, effectively partitioning the dataset into "clean" and "noisy" subsets.
The final training objective is a hybrid: standard Cross-Entropy loss is applied to the clean subset to maximize precision, while the robust MAE loss is applied to the noisy subset to prevent overfitting to incorrect labels.
Theoretical and Empirical Insights
A critical finding of this work is the "redemption" of MAE loss within the prompt learning paradigm. As shown in the experimental results, MAE loss maintains stable accuracy across varying noise rates, whereas CE loss performance degrades significantly as noise increases.
The theoretical analysis suggests that MAE ensures task-relevant features dominate the learning process even when labels are flipped, effectively filtering out the "signal" from the "noise."
Experimental Results
NLPrompt was evaluated on seven synthetic noise benchmarks and the real-world noisy dataset Food101N.
- Extreme Noise Resilience: On the Flowers102 dataset with 75% noise, NLPrompt maintained 76.80% accuracy, while the standard CoOp method plummeted to 37.17%.
- Real-World Application: On the naturally noisy Food101N dataset, NLPrompt reached 76.46% accuracy, outperforming the previous state-of-the-art method, JoAPR, by nearly 4%.
- Versatility: The PromptOT purification method was shown to improve other prompt-tuning architectures, including VPT, MaPLe, and PromptSRC.
Conclusion
NLPrompt provides a simple yet powerful solution for robust prompt learning in vision-language models. By combining the theoretical robustness of MAE loss with a purification strategy that exploits the internal knowledge of pre-trained VLMs, it achieves state-of-the-art results on both synthetic and real-world noisy datasets. Future work may explore extending this framework to handle long-tailed class distributions and more complex generative tasks.
Summary updated