Debiased Orthogonal Boundary-driven Efficient Noise Mitigation

Hao Li, Jiayang Gu, Jingkuan Song, An Zhang, Lianli GaoSubmission to ICML 2025 · Rejected

TL;DR

Web-scale datasets are riddled with mismatched and noisy labels, but existing noise-mitigation techniques are computationally heavy and tightly coupled to specific architectures. This paper introduces One-Step Anti-noise (OSA), a model-agnostic approach that uses an off-the-shelf multimodal foundation model (like CLIP) as an external estimator to evaluate sample cleanliness in a single forward pass. By mathematically characterizing and correcting for the positive shift caused by the "cone effect" in embedding spaces, OSA eliminates noise with negligible training overhead.

What Problem Does This Paper Address?

Training foundation models on billions of web-scraped image-text pairs offers massive scale, but web data is notoriously noisy. Existing methods for learning with noisy labels suffer from three main drawbacks:

  1. High computational cost: Most state-of-the-art approaches rely on dual-network co-training, iterative sample filtration, or multiple backward passes per training step, effectively doubling training time.
  2. Model and task dependency: Many techniques are hand-crafted for specific architectures (e.g., standard CNNs) or specific tasks (such as pure classification), making them hard to adapt across diverse multimodal pipelines.
  3. Imprecise decision boundaries: While pre-trained models can act as zero-shot noise detectors, their high-dimensional cosine similarity space is anisotropic. In theory, two unrelated vectors should be orthogonal ($\cos \approx 0$). In practice, the empirical intersection between clean and noisy pairs deviates significantly from zero, leading to naive filtering thresholds that discard useful data or retain false positives.

Methods

OSA decouples noise estimation from target model training using a one-step inference procedure:

[Input Pair (x, y)] ───► [Estimator (CLIP)] ───► Cosine Similarity s(x, y)
                                                        │
                                                        ▼
[Target Model Loss] ◄─── Loss Weight w ◄─── [Debias: s - β & Score Function]
  • Spatial Debiasing: Deep representations naturally suffer from the cone effect, where embeddings collapse into a narrow cone and shift the orthogonal boundary from $0$ to a positive offset $\beta$. OSA estimates $\beta$ simply by calculating the average cosine similarity of $K$ random vector pairs through the estimator ($\beta \approx 0.215$ for CLIP, $0.087$ for ALIGN).
  • High-Degree Scoring Function: Using the debiased similarity $\tilde{s}_{x,y} = s(x, y) - \beta$, OSA calculates a soft sample weight $w \in [0, 1]$: $$w(x, y, \beta) = \begin{cases} 0 & \text{if } \tilde{s}_{x,y} \le 0 \\ -(\tilde{s}_{x,y})^2(\tilde{s}_{x,y} - 1) & \text{otherwise} \end{cases}$$ Pairs below the shifted boundary are zeroed out as noise, while pairs above it receive weights with rapid gradient transitions near the boundary to prevent ambiguous samples from distorting training.

  • Target-Agnostic Re-weighting: The target model trains on the re-weighted loss $\mathcal{L}_{re} = w \cdot \mathcal{L}_{x,y}$. No extra backward passes or dual-model synchronizations are required.

Datasets

The authors evaluated OSA across multimodal and vision-only benchmarks:

  • MS-COCO & Flickr30K: Standard image-text retrieval datasets evaluated with synthetic noise injected at rates ranging from 20% to 60%.
  • CC120K: A real-world web-crawled multimodal dataset with natural noise estimated between 3% and 20%.
  • SDM Dataset (Synthetic Out-of-Domain): A newly synthesized edge-case dataset created by generating sketch- and cartoon-style variations of MS-COCO images via Latent Diffusion, designed to test estimator boundary stability on unfamiliar domains.
  • WebFG-496 (Aircraft, Bird, Car): Web-scraped fine-grained image classification benchmarks containing real-world label noise.
  • CARS98N: A real-world noisy retrieval benchmark collected from Pinterest (~50% noise).

Key Insights and Ideas

  • Orthogonality Holds, But It Shifts: In high-dimensional spaces, unrelated vectors are almost guaranteed to be orthogonal ($\approx 99.86\%$ fall within $[-0.1, 0.1]$ in 1024 dimensions). Contrastive training drives true pairs positive and false pairs negative. Although neural network layers compress embeddings into a narrow cone, the authors prove (Theorem 2.1) that relative sample relationships remain intact—the boundary simply shifts by a constant offset $\beta$.
  • The Boundary is Dataset-Invariant: The empirical intersection between clean and noisy cosine similarity distributions remains stable across totally different datasets (even on extreme distribution shifts like cartoon/sketch renderings). Thus, $\beta$ is a model-intrinsic property, not a dataset-dependent hyperparameter.
  • Zero-Shot Estimators Suffice: Off-the-shelf zero-shot CLIP estimates noise just as effectively as domain-adapted estimators, removing the need for preliminary warm-up stages.

Key Results

  • Robust Cross-Modal Matching: On MS-COCO 5K under heavy (50%) noise, OSA achieves 64.7% i2t R@1 and 48.6% t2i R@1, outperforming the prior state-of-the-art method NPC (59.9% and 43.0%). Across noise levels from 0% to 50%, OSA's R@1 dropped by only 1.28%, compared to a 5.05% drop for NPC.
  • Massive Recovery on Flickr30K: At 60% noise, OSA improved baseline CLIP R@1 by +20.9% (i2t) and +22.3% (t2i).
  • Broad Task Transfer: In image classification on WebFG-496, OSA boosted accuracy by +7.74% (Aircraft), +8.21% (Bird), and +4.28% (Car). On CARS98N retrieval, it improved Precision@1 by +6.76% and mAP by +6.83%.
  • Order-of-Magnitude Efficiency Gains: On MS-COCO, baseline CLIP training took 97 minutes. NPC took 323 minutes (+226 min overhead due to double backward passes), whereas OSA required only 118 minutes (+21 min overhead). The authors estimate offline inference for 1 billion samples would take ~75 hours on a single RTX 3090.

Limitations and Future Work

  • No Direct Pre-training Validation at Scale: Due to computational resource constraints, the authors could not run an end-to-end pre-training run on billion-scale datasets from scratch; they instead simulated these conditions on CC120K.
  • Future Work: The authors highlight validating OSA directly within billion-scale pre-training pipelines (e.g., CC4M/CC12M or larger web runs) and exploring its domain adaptability across non-visual modalities as key next steps.

Summary updated

Abstract

Mitigating the detrimental effects of noisy labels on the training process has become increasingly critical, as obtaining entirely clean or human-annotated samples for large-scale pre-training tasks is often impractical. Nonetheless, existing noise mitigation methods often encounter limitations in practical applications due to their task-specific design, model dependency, and significant computational overhead. In this work, we exploit the properties of high-dimensional orthogonality to identify a robust and effective boundary in cone space for separating clean and noisy samples. Building on this, we propose One-Step Anti-noise (OSA), a model-agnostic noisy label mitigation paradigm that employs an estimator model and a scoring function to assess the noise level of input pairs through just one-step inference. We empirically validate the superiority of OSA, demonstrating its enhanced training robustness, improved task transferability, streamlined deployment, and reduced computational overhead across diverse benchmarks, models, and tasks. Our code is released at https://anonymous.4open.science/r/CLIP_OSN-E86C.
BibTeX
@misc{li2024debiased,
  author = {Hao Li and Jiayang Gu and Jingkuan Song and An Zhang and Lianli Gao},
  title = {{Debiased Orthogonal Boundary-driven Efficient Noise Mitigation}},
  year = {2024},
  eprint = {2410.01944},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2410.01944},
}
Figure