Screening, Rectifying, and Re-Screening: A Unified Framework for Tuning Vision-Language Models with Noisy Labels

Chaowei Fang, Hangfei Ma, Zhihao Li, De Cheng, Yue Zhang, Guanbin LiIJCAI 2025 · DOI

Presented at IJCAI-25, this research introduces a robust framework for fine-tuning Vision-Language Models (VLMs) when training data contains noisy labels—images paired with incorrect textual descriptions.

Introduction

Fine-tuning pre-trained models like CLIP is the standard approach for adapting them to specific tasks. However, real-world datasets often contain "noisy labels" due to human error or automated scraping. Deep Neural Networks (DNNs) are prone to memorizing these errors, which degrades performance. While the "small-loss criterion" is often used to distinguish clean from noisy data, it frequently fails in complex vision-language tasks where boundaries are blurry. Furthermore, models suffer from self-confirmation bias, where they trust their own incorrect predictions during training, creating a feedback loop of error.

The SRR Framework

The proposed SRR framework addresses these issues through a three-stage pipeline: Screening, Rectifying, and Re-Screening.

The SRR framework consists of three stages: Label Screening based on Dual-level Semantic Matching, Label Rectification, and Re-screening using an auxiliary BLIP evaluator.

1. Screening via Dual-Level Semantic Matching

Instead of relying on a single text prompt, the framework uses two levels of semantic matching:

  • Macro-level: Generic class names to ensure broad category separation.
  • Micro-level: Detailed attribute descriptions (e.g., "a photo of a beagle, which is a friendly and curious breed") to capture fine-grained features.

By comparing losses from these two prompt types, a Gaussian Mixture Model (GMM) partitions data into three segments: Clean, Ambiguous, and Noisy. Empirical evidence shows that dual-level prompts provide a much clearer signal for this partitioning than single-level prompts.

Loss distributions for clean and noisy data across different prompt types. Dual-level prompts provide the clearest separation, facilitating more accurate data partitioning.

2. Label Rectification

Labels are updated based on their assigned segment. Clean samples retain their original labels, while noisy samples are assigned pseudo-labels generated by the model. Ambiguous samples receive a weighted combination of both, acknowledging the inherent uncertainty in the data.

3. Re-Screening and Validation

To break the self-confirmation bias, the authors introduce an auxiliary VLM, BLIP, to cross-validate the rectified labels. If BLIP confirms a high-quality match, the sample is used for standard prompt learning. If not, the sample undergoes mixup-based augmentation to minimize the impact of potential remaining errors.

Experimental Evaluation

The framework was evaluated on ten diverse datasets, including ImageNet, OxfordPets, and EuroSAT, covering general objects, fine-grained categories, and specialized imagery. Testing included both Symflip (random label swapping) and Pairflip (swapping with similar classes) noise, with rates up to 75%.

Key Results

  • State-of-the-Art Performance: SRR consistently outperformed existing methods like CoOp and JoAPR. On the DTD dataset with 62.5% Symflip noise, SRR improved accuracy by 8.73% over the previous best.
  • Extreme Robustness: In a stress test with 100% label noise on Caltech101, the model still achieved 87.71% accuracy, whereas the baseline CoOp model collapsed to near 1%.
  • Consistency: Across all datasets and noise levels, the SRR framework maintained the highest average accuracy, demonstrating its generalizability.

Conclusion

The SRR framework provides a comprehensive solution for tuning VLMs in the presence of noisy labels. By leveraging dual-level semantic insights and independent cross-model validation, it effectively mitigates the risks of label noise and self-confirmation bias, establishing a new state-of-the-art for robust vision-language learning.

Summary updated

BibTeX
@inproceedings{fang2025screening,
  author = {Chaowei Fang and Hangfei Ma and Zhihao Li and De Cheng and Yue Zhang and Guanbin Li},
  title = {{Screening, Rectifying, and Re-Screening: A Unified Framework for Tuning Vision-Language Models with Noisy Labels}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.24963/ijcai.2025/568},
}
Figure