One-step Noisy Label Mitigation

Hao Li, Jiayang Gu, Jingkuan Song, An Zhang, Lianli Gao

Abstract

Large-scale multimodal training often relies on noisy web-scraped datasets, which degrades performance and increases computational costs. We present One-Step Anti-noise (OSA), a model-agnostic framework that identifies and down-weights noisy samples in a single inference step. By leveraging the high-dimensional geometry of pre-trained models and correcting for the "cone effect" bias, OSA achieves state-of-the-art results across various benchmarks with significantly reduced overhead compared to existing multi-step methods.

Introduction

As machine learning scales, the reliance on massive web-scraped datasets has introduced significant challenges due to data noise. These datasets often contain mismatched pairs, such as an image of a dog incorrectly labeled as a cat. Training on such noise degrades model performance and wastes computational resources.

Existing noise mitigation strategies typically suffer from three major drawbacks: they are often task-specific, tightly coupled with specific model architectures, and computationally expensive, frequently requiring multiple training passes that can double or triple training time. OSA addresses these issues by providing a "plug-and-play" solution that is efficient, generalizable, and easily integrated into existing training pipelines.

Methodology

OSA utilizes an external Estimator Model (e.g., a frozen, pre-trained model like CLIP or ALIGN) to evaluate the cleanliness of data pairs. The framework operates in three primary stages:

  1. Spatial Debiasing: High-dimensional embeddings often cluster into a narrow anisotropic cone, shifting the theoretical "zero point" of cosine similarity. OSA calculates a shift factor ($\beta$) to re-center the coordinate system, correcting for this "cone effect."
  2. Noise Scoring: Debiased similarity scores are processed through a high-degree scoring function. This function aggressively down-weights samples below the orthogonal boundary (likely noise) while rewarding high-confidence clean samples.
  3. Re-weighted Training: During training, the loss for each sample is multiplied by its OSA-generated weight ($w$). This ensures that noisy samples contribute minimally to the gradient, preventing model corruption.
The OSA framework consists of a Scoring Phase to determine sample weights and a Training Phase where these weights are applied to the loss function.

Crucially, OSA requires only a single inference step from the estimator, avoiding the high costs associated with dual-model or multi-pass approaches.

Experimental Evaluation

The authors evaluated OSA across a diverse set of benchmarks, including cross-modal matching (MSCOCO, Flickr30K), real-world noisy data (CC120K, CC3M), image classification (WebFG-496), and image retrieval (CARS98N). To test robustness, they also utilized a custom SDM dataset featuring images in uncommon artistic styles.

Key Results

OSA demonstrates superior performance and efficiency compared to existing state-of-the-art (SOTA) methods:

  • Performance Gains: On MSCOCO with 50% noise, OSA (using an ALIGN estimator) outperformed the previous SOTA (NPC) by 8.6% in image-to-text and 7.0% in text-to-image retrieval (R@1).
  • Efficiency: OSA reduced additional training time overhead by 90% compared to NPC, adding only 21 minutes in a benchmark where NPC added 226 minutes.
  • Noise Robustness: While competitors' performance often drops significantly as noise increases, OSA showed remarkable stability, with only a 1.275% decrease in performance when moving from 0% to 50% noise.
Performance comparison on MS-COCO 1K and 5K datasets across various noise ratios (0%, 20%, 50%). OSA consistently improves performance over baselines and outperforms other noise mitigation methods.

Discussion and Conclusion

The success of OSA stems from the observation that the intersection point between clean and noisy distributions remains consistent for a given model, suggesting the boundary is a property of the embedding space rather than the data. This allows for effective filtering based on geometric orthogonality. Furthermore, the architecture-agnostic nature of OSA allows it to be applied to various target models, including ResNet, VGG, and Vision Transformers.

While highly effective, future work includes evaluating OSA on billion-scale pre-training runs and exploring its application in other modalities such as video-audio or medical imaging. OSA represents a significant step toward efficient and robust training on imperfect, large-scale web data.

Summary updated

Abstract

Mitigating the detrimental effects of noisy labels on the training process has become increasingly critical, as obtaining entirely clean or human-annotated samples for large-scale pre-training tasks is often impractical. Nonetheless, existing noise mitigation methods often encounter limitations in practical applications due to their task-specific design, model dependency, and significant computational overhead. In this work, we exploit the properties of high-dimensional orthogonality to identify a robust and effective boundary in cone space for separating clean and noisy samples. Building on this, we propose One-Step Anti-noise (OSA), a model-agnostic noisy label mitigation paradigm that employs an estimator model and a scoring function to assess the noise level of input pairs through just one-step inference. We empirically validate the superiority of OSA, demonstrating its enhanced training robustness, improved task transferability, streamlined deployment, and reduced computational overhead across diverse benchmarks, models, and tasks. Our code is released at https://anonymous.4open.science/r/CLIP_OSN-E86C.
BibTeX
@misc{li2024one,
  author = {Hao Li and Jiayang Gu and Jingkuan Song and An Zhang and Lianli Gao},
  title = {{One-step Noisy Label Mitigation}},
  year = {2024},
  eprint = {2410.01944},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2410.01944},
}
Figure