Self-Cleaning: Improving a Named Entity Recognizer Trained on Noisy Data with a Few Clean Instances

Zhendong Chu, Ruiyi Zhang, Tong Yu, Rajiv Jain, Vlad I. Morariu, Jiuxiang Gu, Ani Nenkova

Abstract

To achieve state-of-the-art performance, one still needs to train NER models on large-scale, high-quality annotated data, an asset that is both costly and time-intensive to accumulate. In contrast, real-world applications often resort to massive low-quality labeled data through non-expert annotators via crowdsourcing and external knowledge bases via distant supervision as a cost-effective alternative. However, these annotation methods result in noisy labels, which in turn lead to a notable decline in performance. Hence, we propose to denoise the noisy NER data with guidance from a small set of clean instances. Along with the main NER model we train a discriminator model and use its outputs to recalibrate the sample weights. The discriminator is capable of detecting both span and category errors with different discriminative prompts. Results on public crowdsourcing and distant supervision datasets show that the proposed method can consistently improve performance with a small guidance set.

Introduction

Training high-performance Named Entity Recognition (NER) models typically requires massive amounts of high-quality, human-annotated data. However, obtaining such data is expensive and time-consuming. Consequently, many real-world applications rely on "noisy" alternatives like crowdsourcing or distant supervision. These methods often introduce two major types of noise: span errors (incorrect boundaries) and category errors (incorrect entity types).

Comparison between previous noisy supervision and the proposed Self-Cleaning method using a guidance set.

While self-training is a common approach to mitigate noise, it often suffers from "confirmation bias," where the model reinforces its own initial mistakes. The Self-Cleaning framework addresses this by using a very small "guidance set" of clean data to effectively denoise large-scale noisy labels, breaking the error-propagation loop.

Methodology

The Self-Cleaning framework operates in three distinct stages to improve model robustness and denoising accuracy.

The two-stage Self-Cleaning framework: Demonstrative Self-training and Co-training with a prompt-based discriminator.

Stage I: Demonstrative Self-Training

In this stage, the model receives "demonstrations" from the clean guidance set. By prepending similar clean examples to the input (e.g., [SEP] San Jose is a location), the model is nudged toward more robust patterns before processing the noisy training data.

Stage II: Co-training with a Prompt-based Discriminator

The core innovation is an auxiliary RoBERTa-based discriminator trained on the small clean set. Instead of a simple classifier, it uses Masked Language Model (MLM) prompts to detect errors:

  • Span Prompt: [Sentence]. [Entity] is a [MASK] entity.
  • Category Prompt: [Sentence]. [Entity] is a [MASK] [Type] entity.

The discriminator's confidence scores are used to reweight noisy training samples. If the discriminator identifies a label as likely incorrect, its influence on the main NER model is downweighted.

Stage III: Fine-tuning

Finally, the denoised model is fine-tuned directly on the small guidance set to further polish its performance and align it with the gold-standard labels.

Experiments and Results

The framework was evaluated on several standard NER benchmarks, including CoNLL03-C (crowdsourced), and distantly supervised versions of CoNLL03, OntoNotes 5.0, and Wikigold.

Key Findings

  • State-of-the-Art Performance: On the CoNLL03-C dataset, Self-Cleaning achieved an F1 score of 0.862, significantly outperforming previous baselines like RoSTER (0.804) and NEEDLE (0.842).
  • Sample Efficiency: The method is remarkably efficient; it reached an F1 score of 0.880 using only 200 clean instances, whereas a standard model required 1,000 clean instances to achieve the same performance level.
  • Handling Span Errors: Ablation studies showed that the "Span Discriminator" is critical. Roughly 11-13% of entities in noisy datasets have imperfect spans, and specifically targeting these boundary issues leads to significant gains.

Conclusion and Future Work

Self-Cleaning demonstrates that a small amount of high-quality data can go a long way in denoising massive low-quality datasets. However, the current discriminator struggles with "non-entity" (O) labels, potentially missing entities entirely (false negatives). Future work could explore token-level weighting to salvage correct parts of partially wrong spans and the use of more sophisticated "hard negatives" to further sharpen the discriminator's accuracy.

Summary updated

Abstract

To achieve state-of-the-art performance, one still needs to train NER models on large-scale, high-quality annotated data, an asset that is both costly and time-intensive to accumulate. In contrast, real-world applications often resort to massive low-quality labeled data through non-expert annotators via crowdsourcing and external knowledge bases via distant supervision as a cost-effective alternative. However, these annotation methods result in noisy labels, which in turn lead to a notable decline in performance. Hence, we propose to denoise the noisy NER data with guidance from a small set of clean instances. Along with the main NER model we train a discriminator model and use its outputs to recalibrate the sample weights. The discriminator is capable of detecting both span and category errors with different discriminative prompts. Results on public crowdsourcing and distant supervision datasets show that the proposed method can consistently improve performance with a small guidance set.
BibTeX
@inproceedings{chu2024self,
  author = {Zhendong Chu and Ruiyi Zhang and Tong Yu and Rajiv Jain and Vlad I. Morariu and Jiuxiang Gu and Ani Nenkova},
  title = {{Self-Cleaning: Improving a Named Entity Recognizer Trained on Noisy Data with a Few Clean Instances}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
  url = {https://openreview.net/forum?id=AIfmOITgFN},
}
Figure