Abstract
The emergence of vision-language foundation models, such as CLIP, has revolutionized image-text representation, enabling a broad range of applications via prompt learning. Despite its promise, real-world datasets often contain noisy labels that can degrade prompt learning performance. In this paper, we demonstrate that using mean absolute error (MAE) loss in prompt learning, named PromptMAE, significantly enhances robustness against noisy labels while maintaining high accuracy. Though MAE is straightforward and recognized for its robustness, it is rarely used in noisy-label learning due to its slow convergence and poor performance outside prompt learning scenarios. To elucidate the robustness of PromptMAE, we leverage feature learning theory to show that MAE can suppress the influence of noisy samples, thereby improving the signal-to-noise ratio and enhancing overall robustness. Additionally, we introduce PromptOT, a prompt-based optimal transport data purification method to enhance the robustness further. PromptOT employs text features in vision-language models as prototypes to construct an optimal transportation matrix. This matrix effectively partitions datasets into clean and noisy subsets, allowing for the application of cross-entropy loss to the clean subset and MAE loss to the noisy subset. Our Noise-Label Prompt Learning method, named NLPrompt, offers a simple and efficient approach that leverages the expressive representations and precise alignment capabilities of vision-language models for robust prompt learning. We validate NLPrompt through extensive experiments across various noise settings, demonstrating significant performance improvements.
Introduction
Prompt learning has emerged as a highly efficient paradigm for adapting large-scale vision-language models (VLMs) like CLIP to downstream tasks. By optimizing only a small set of continuous prompt vectors while keeping the backbone frozen, researchers can achieve competitive performance with minimal data. However, a significant hurdle remains: real-world datasets are frequently plagued by noisy labels. Standard prompt-tuning methods typically rely on Cross-Entropy (CE) loss, which is highly sensitive to label noise, often causing the model to overfit to incorrect labels and lose generalization capability.
Methodology: The NLPrompt Framework
To address the challenge of noisy labels, we propose NLPrompt, a dual-strategy framework that combines data purification with robust loss functions. The architecture is centered around two primary innovations: PromptOT and PromptMAE.
PromptOT: Optimal Transport for Data Purification
Unlike traditional data purification methods that rely on local thresholds or random initializations, PromptOT leverages the inherent knowledge of the pre-trained VLM. It uses the model's own text features as prototypes to solve a global optimal transport problem. By constructing a cost matrix between image and text features, PromptOT identifies the most likely assignments for each sample. If the assigned pseudo-label matches the dataset's given label, the sample is categorized into a "clean" subset ($D_{clean}$); otherwise, it is moved to a "noisy" subset ($D_{noisy}$).
PromptMAE: Robustness via Mean Absolute Error
For the noisy subset, we employ PromptMAE, utilizing Mean Absolute Error (MAE) loss. While MAE is often avoided in deep learning due to slow convergence, we demonstrate that in the low-parameter regime of prompt tuning, MAE acts as a powerful regularizer. Theoretical analysis using feature learning theory suggests that MAE suppresses the growth of task-irrelevant (noisy) features, effectively improving the signal-to-noise ratio during training.
Experimental Results
We evaluated NLPrompt across seven benchmark datasets (e.g., Flowers102, Caltech101, UCF101) under various synthetic noise conditions, as well as on the naturally noisy Food101N dataset.
Performance Under High Noise
NLPrompt demonstrates remarkable resilience in extreme noise scenarios. As shown in the comprehensive results table below, our method consistently outperforms baselines like CoOp, GCE, and JoAPR. For instance, on the Flowers102 dataset with a 75% symmetric noise rate, NLPrompt maintains an accuracy of 76.80%, whereas the standard CoOp baseline drops to 37.17%.
Real-World and Plug-and-Play Utility
On the real-world noisy dataset Food101N, NLPrompt achieved an accuracy of 76.46%, surpassing the previous state-of-the-art. Furthermore, the NLPrompt strategy is model-agnostic; when integrated into other prompt learning architectures such as VPT, MaPLe, and PromptSRC, it consistently yielded performance gains of 10-40% in high-noise environments.
Conclusion
NLPrompt provides a simple yet effective solution for robust prompt learning in vision-language models. By utilizing the pre-aligned representations of VLMs for data purification (PromptOT) and the inherent robustness of MAE in low-parameter settings (PromptMAE), we bridge the gap between efficient adaptation and real-world data quality challenges. Future work will explore the application of NLPrompt to highly unbalanced or long-tailed noisy datasets.
Summary updated