NoiseBench: Benchmarking the Impact of Real Label Noise on Named Entity Recognition

Elena Merdjanovska, Ansar Aynetdinov, Alan AkbikEMNLP 2024 · DOI

Abstract

Supervised learning for Named Entity Recognition (NER) typically assumes high-quality training data, yet real-world datasets often contain significant label noise. While noise-robust algorithms exist, they are predominantly evaluated on simulated, random label flips. This paper introduces NoiseBench, a benchmark featuring six types of real-world noise—ranging from expert errors to LLM-generated labels. Our findings indicate that real-world noise is substantially more challenging than simulated noise because models tend to memorize structured, plausible errors almost immediately, bypassing the typical generalization phase seen in synthetic setups.

Introduction

Named Entity Recognition (NER) is a fundamental task in natural language processing, identifying entities such as locations (LOC), organizations (ORG), and miscellaneous entities (MISC). High-performance models require large, accurately labeled datasets. However, even gold-standard datasets like CoNLL-03 are estimated to contain 5% to 7% noise.

To address this, researchers have developed various noise-robust learning techniques. Traditionally, these are tested using simulated noise, where labels are randomly flipped. The authors of NoiseBench argue that this "toy" setup fails to capture the complexity of real-world errors. Unlike random flips, real noise—stemming from human fatigue, ambiguous guidelines, or automated heuristics—is often structured and linguistically plausible.

The NoiseBench Framework

The authors developed NoiseBench by creating seven variants of the English and German CoNLL-03 training sets: one clean version and six noisy versions.

Types of Real-World Noise

  1. Expert Errors: Original, uncorrected labels from the initial CoNLL-03 release.
  2. Crowdsourcing: Labels derived from majority voting among 47 crowd workers.
  3. Distant Supervision: Entity matching against knowledge bases like Wikidata.
  4. Weak Supervision: A combination of rule-based heuristics.
  5. LLM Teacher: Labels generated by prompting GPT-3.5.
Example of a correctly labeled sentence from the CoNLL-03 dataset.

In contrast to the clean labels above, real-world noise often introduces plausible but incorrect classifications or over-annotations that are difficult for models to distinguish from ground truth.

Example of real-world noise from crowdsourcing, showing a plausible misclassification ("Swiss" as LOC) and an over-annotation ("Sunday" as MISC).

Key Insights: Why Real Noise is Harder

The study reveals a critical "robustness gap." While models training on simulated noise often exhibit a "generalization phase"—learning correct patterns before memorizing noise—this phase is absent with real noise.

Plausibility and Memorization

Real noise is often "plausible." For instance, labeling "Swiss" as a Location (LOC) instead of a Miscellaneous (MISC) entity is a common human error. Because these errors follow linguistic patterns, deep learning models treat them as valid signals rather than outliers, leading to immediate memorization from the first epoch. This structured nature makes it nearly impossible for standard noise-robust methods to filter out errors effectively.

The LLM Factor

Interestingly, the study found that GPT-3.5, often used as a "silver" labeler, produced the highest noise share (45.6%). This was primarily due to "hallucinations," where the model identified common nouns as named entities, further complicating the training process for downstream models.

Evaluation of Robust Methods

The authors benchmarked several state-of-the-art (SOTA) robust learning methods, including Confident Learning (filtering likely noisy samples), BOND (teacher-student pseudo-labeling), and MSR (Meta Self-Refinement).

Results

  • Performance Degradation: Models trained on real noise saw a test F1 score drop of approximately 2.5 percentage points compared to those trained on simulated noise of equal intensity.
  • Method Efficacy: No single method dominated across all noise types. MSR performed best for English, while Confident Learning was more effective for German.
  • The Oracle Subset: The most successful strategy was identifying a perfectly clean subset of the data. Most robust methods still fall short of this "Oracle" upper bound, suggesting that current algorithms are better at handling random noise than structured real-world errors.

Conclusion and Future Work

NoiseBench demonstrates that current noise-robust algorithms are over-optimized for synthetic noise and struggle with the structured errors found in real-world data. Future research must focus on developing algorithms that can specifically distinguish between "plausible but wrong" labels and ground truth. The benchmark serves as a call to action for the community to move beyond toy simulations toward more realistic data challenges.

Summary updated

BibTeX
@inproceedings{merdjanovska2024noisebench,
  author = {Elena Merdjanovska and Ansar Aynetdinov and Alan Akbik},
  title = {{NoiseBench: Benchmarking the Impact of Real Label Noise on Named Entity Recognition}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2024},
  doi = {10.18653/v1/2024.emnlp-main.1011},
}
Figure