Presented at IJCAI-25, this research introduces a robust framework for fine-tuning Vision-Language Models (VLMs) when training data contains noisy labels—images paired with incorrect textual descriptions.
Introduction
Fine-tuning pre-trained models like CLIP is the standard approach for adapting them to specific tasks. However, real-world datasets often contain "noisy labels" due to human error or automated scraping. Deep Neural Networks (DNNs) are prone to memorizing these errors, which degrades performance. While the "small-loss criterion" is often used to distinguish clean from noisy data, it frequently fails in complex vision-language tasks where boundaries are blurry. Furthermore, models suffer from self-confirmation bias, where they trust their own incorrect predictions during training, creating a feedback loop of error.
The SRR Framework
The proposed SRR framework addresses these issues through a three-stage pipeline: Screening, Rectifying, and Re-Screening.
1. Screening via Dual-Level Semantic Matching
Instead of relying on a single text prompt, the framework uses two levels of semantic matching:
- Macro-level: Generic class names to ensure broad category separation.
- Micro-level: Detailed attribute descriptions (e.g., "a photo of a beagle, which is a friendly and curious breed") to capture fine-grained features.
By comparing losses from these two prompt types, a Gaussian Mixture Model (GMM) partitions data into three segments: Clean, Ambiguous, and Noisy. Empirical evidence shows that dual-level prompts provide a much clearer signal for this partitioning than single-level prompts.
2. Label Rectification
Labels are updated based on their assigned segment. Clean samples retain their original labels, while noisy samples are assigned pseudo-labels generated by the model. Ambiguous samples receive a weighted combination of both, acknowledging the inherent uncertainty in the data.
3. Re-Screening and Validation
To break the self-confirmation bias, the authors introduce an auxiliary VLM, BLIP, to cross-validate the rectified labels. If BLIP confirms a high-quality match, the sample is used for standard prompt learning. If not, the sample undergoes mixup-based augmentation to minimize the impact of potential remaining errors.
Experimental Evaluation
The framework was evaluated on ten diverse datasets, including ImageNet, OxfordPets, and EuroSAT, covering general objects, fine-grained categories, and specialized imagery. Testing included both Symflip (random label swapping) and Pairflip (swapping with similar classes) noise, with rates up to 75%.
Key Results
- State-of-the-Art Performance: SRR consistently outperformed existing methods like CoOp and JoAPR. On the DTD dataset with 62.5% Symflip noise, SRR improved accuracy by 8.73% over the previous best.
- Extreme Robustness: In a stress test with 100% label noise on Caltech101, the model still achieved 87.71% accuracy, whereas the baseline CoOp model collapsed to near 1%.
- Consistency: Across all datasets and noise levels, the SRR framework maintained the highest average accuracy, demonstrating its generalizability.
Conclusion
The SRR framework provides a comprehensive solution for tuning VLMs in the presence of noisy labels. By leveraging dual-level semantic insights and independent cross-model validation, it effectively mitigates the risks of label noise and self-confirmation bias, establishing a new state-of-the-art for robust vision-language learning.
Summary updated