Abstract
Large-scale multimodal training often relies on noisy web-scraped datasets, which degrades performance and increases computational costs. We present One-Step Anti-noise (OSA), a model-agnostic framework that identifies and down-weights noisy samples in a single inference step. By leveraging the high-dimensional geometry of pre-trained models and correcting for the "cone effect" bias, OSA achieves state-of-the-art results across various benchmarks with significantly reduced overhead compared to existing multi-step methods.
Introduction
As machine learning scales, the reliance on massive web-scraped datasets has introduced significant challenges due to data noise. These datasets often contain mismatched pairs, such as an image of a dog incorrectly labeled as a cat. Training on such noise degrades model performance and wastes computational resources.
Existing noise mitigation strategies typically suffer from three major drawbacks: they are often task-specific, tightly coupled with specific model architectures, and computationally expensive, frequently requiring multiple training passes that can double or triple training time. OSA addresses these issues by providing a "plug-and-play" solution that is efficient, generalizable, and easily integrated into existing training pipelines.
Methodology
OSA utilizes an external Estimator Model (e.g., a frozen, pre-trained model like CLIP or ALIGN) to evaluate the cleanliness of data pairs. The framework operates in three primary stages:
- Spatial Debiasing: High-dimensional embeddings often cluster into a narrow anisotropic cone, shifting the theoretical "zero point" of cosine similarity. OSA calculates a shift factor ($\beta$) to re-center the coordinate system, correcting for this "cone effect."
- Noise Scoring: Debiased similarity scores are processed through a high-degree scoring function. This function aggressively down-weights samples below the orthogonal boundary (likely noise) while rewarding high-confidence clean samples.
- Re-weighted Training: During training, the loss for each sample is multiplied by its OSA-generated weight ($w$). This ensures that noisy samples contribute minimally to the gradient, preventing model corruption.
Crucially, OSA requires only a single inference step from the estimator, avoiding the high costs associated with dual-model or multi-pass approaches.
Experimental Evaluation
The authors evaluated OSA across a diverse set of benchmarks, including cross-modal matching (MSCOCO, Flickr30K), real-world noisy data (CC120K, CC3M), image classification (WebFG-496), and image retrieval (CARS98N). To test robustness, they also utilized a custom SDM dataset featuring images in uncommon artistic styles.
Key Results
OSA demonstrates superior performance and efficiency compared to existing state-of-the-art (SOTA) methods:
- Performance Gains: On MSCOCO with 50% noise, OSA (using an ALIGN estimator) outperformed the previous SOTA (NPC) by 8.6% in image-to-text and 7.0% in text-to-image retrieval (R@1).
- Efficiency: OSA reduced additional training time overhead by 90% compared to NPC, adding only 21 minutes in a benchmark where NPC added 226 minutes.
- Noise Robustness: While competitors' performance often drops significantly as noise increases, OSA showed remarkable stability, with only a 1.275% decrease in performance when moving from 0% to 50% noise.
Discussion and Conclusion
The success of OSA stems from the observation that the intersection point between clean and noisy distributions remains consistent for a given model, suggesting the boundary is a property of the embedding space rather than the data. This allows for effective filtering based on geometric orthogonality. Furthermore, the architecture-agnostic nature of OSA allows it to be applied to various target models, including ResNet, VGG, and Vision Transformers.
While highly effective, future work includes evaluating OSA on billion-scale pre-training runs and exploring its application in other modalities such as video-audio or medical imaging. OSA represents a significant step toward efficient and robust training on imperfect, large-scale web data.
Summary updated