Neighborhood Collective Estimation for Noisy Label Identification and Correction

Jichang Li, Guanbin Li, Feng Liu, Yizhou Yu

Abstract

Deep neural networks are highly susceptible to memorizing incorrect labels, a phenomenon that significantly degrades performance in real-world applications where data is often noisy. Traditional methods for Learning with Noisy Labels (LNL) typically rely on a model's own predictions to filter noise, which often results in confirmation bias—a feedback loop where early errors are reinforced. This paper introduces Neighborhood Collective Estimation (NCE), a framework that leverages the feature-space proximity of samples to identify and correct label noise. By shifting from self-reliant predictions to a neighborhood-based consensus, NCE achieves state-of-the-art performance across synthetic and large-scale real-world datasets, particularly in high-noise scenarios.

Introduction

The efficacy of deep learning is often predicated on the availability of large-scale, accurately labeled datasets. However, obtaining perfect labels is frequently impractical due to the costs of expert annotation or the inherent noise in web-scraped data. When trained on such "messy" data, neural networks tend to memorize noisy labels, leading to poor generalization.

The primary challenge in Learning with Noisy Labels (LNL) is distinguishing between "clean" samples (correctly labeled) and "noisy" samples (incorrectly labeled). Most current approaches use the model's own output to make this distinction. However, this creates a "confirmation bias" problem: if a model is initially uncertain or incorrect about a sample, it may discard a valid but difficult sample as noise, or conversely, confidently memorize a wrong label. NCE addresses this by introducing a collective estimation strategy that uses a sample's neighbors in the feature space as a more objective reference point.

The NCE Framework

NCE operates on the principle that while a model might be biased regarding a specific instance, it is unlikely to be biased in the same way across all similar instances. The framework consists of two primary stages: verification and correction.

1. Neighborhood Collective Noise Verification (NCNV)

To determine if a sample $(x, y)$ is clean, NCE evaluates its consistency relative to its $K$ nearest neighbors in the model's feature space. The system calculates a "label inconsistency score" using Jensen-Shannon (JS) divergence. This score compares the sample's provided label against the predicted class distributions of its neighbors. If the sample's label is a significant outlier compared to the collective prediction of its neighborhood, it is flagged as noisy. This approach is particularly effective at preserving "hard" samples—clean images that are difficult to classify but are surrounded by neighbors of the same semantic class.

2. Neighborhood Collective Label Correction (NCLC)

Once noisy samples are identified, NCE attempts to recover their utility rather than simply discarding them. The framework generates new pseudo-labels by finding the nearest clean neighbors and applying a weighted combination of their ground-truth labels. This creates a stable consensus that is less susceptible to the fluctuations of a single model's instantaneous prediction.

Training Regularization

To further enhance robustness, the model is fine-tuned using Mixup regularization, which encourages the model to learn smoother decision boundaries and prevents the memorization of specific noisy instances. Additionally, consistency regularization is applied to ensure the model produces stable outputs for various perturbed versions of the same image.

Experimental Evaluation

The authors evaluated NCE against several state-of-the-art (SOTA) benchmarks, including DivideMix and ELR, across both synthetic and real-world datasets.

Synthetic Noise (CIFAR-10 & CIFAR-100)

On the CIFAR-10 dataset with 90% symmetric noise—an extreme scenario where 9 out of 10 labels are wrong—NCE achieved an accuracy of 88.4%. This represents a significant improvement over the previous SOTA, DivideMix, which achieved 76.0% under the same conditions. This demonstrates NCE's exceptional robustness in environments with very low signal-to-noise ratios.

Real-World Noise (Clothing-1M & Webvision)

In real-world tests, where noise is often structured and non-random, NCE continued to excel:

  • Clothing-1M: On this dataset of 1 million web-scraped clothing images, NCE reached 75.3% accuracy, setting a new performance benchmark.
  • Webvision-1.0: NCE achieved a 79.5% Top-1 accuracy, outperforming the previous leading method (NGC) by 0.3%.

Discussion and Insights

The success of NCE highlights the value of neighborhood-based context in deep learning. By using feature-space neighbors as a "second opinion," the model can effectively navigate the pitfalls of confirmation bias. This is especially vital for "hard" classes (e.g., distinguishing between similar breeds of dogs), where individual predictions are most likely to fail but neighborhood clusters remain semantically coherent.

However, the method is not without limitations. The reliance on $K$-nearest neighbor (KNN) searches introduces computational overhead, particularly as dataset sizes scale. While efficient similarity search libraries like FAISS can mitigate this, it remains a factor in training time. Furthermore, the method's success is tied to the quality of the initial feature representations; if the model's early "warm-up" phase fails to cluster similar items together, the neighborhood consensus will be inherently flawed.

Conclusion

Neighborhood Collective Estimation provides a robust and intuitive solution to the problem of noisy labels. By leveraging the collective intelligence of feature-space neighbors, NCE offers a path toward training highly accurate models on imperfect data, bridging the gap between controlled academic benchmarks and the messy reality of large-scale data collection.

Summary updated

BibTeX
@misc{li2022neighborhood,
  author = {Jichang Li and Guanbin Li and Feng Liu and Yizhou Yu},
  title = {{Neighborhood Collective Estimation for Noisy Label Identification and Correction}},
  year = {2022},
  eprint = {2208.03207},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2208.03207},
}
Figure