Improving a Named Entity Recognizer Trained on Noisy Data with a Few Clean Instances

Zhendong Chu, Ruiyi Zhang, Tong Yu, Rajiv Jain, Vlad I. Morariu, Jiuxiang Gu, Ani Nenkova

Abstract

To achieve state-of-the-art performance, one still needs to train NER models on large-scale, high-quality annotated data, an asset that is both costly and time-intensive to accumulate. In contrast, real-world applications often resort to massive low-quality labeled data through non-expert annotators via crowdsourcing and external knowledge bases via distant supervision as a cost-effective alternative. However, these annotation methods result in noisy labels, which in turn lead to a notable decline in performance. Hence, we propose to denoise the noisy NER data with guidance from a small set of clean instances. Along with the main NER model we train a discriminator model and use its outputs to recalibrate the sample weights. The discriminator is capable of detecting both span and category errors with different discriminative prompts. Results on public crowdsourcing and distant supervision datasets show that the proposed method can consistently improve performance with a small guidance set.

Introduction

Named Entity Recognition (NER) is a fundamental task in Natural Language Processing, identifying entities like organizations, locations, and persons within text. While deep learning models have achieved impressive results, they typically require vast amounts of high-quality human-annotated data. To mitigate the high cost of manual labeling, practitioners often use crowdsourcing or distant supervision. However, these methods introduce significant noise—specifically span errors (incorrect boundaries) and category errors (incorrect entity types)—which degrade model performance.

The Self-Cleaning Framework

The proposed Self-Cleaning framework addresses these challenges by utilizing a small "guidance set" of clean data to denoise large-scale noisy datasets. The framework operates in three stages:

  1. Stage I: Demonstrative Self-training: The model incorporates clean examples from the guidance set as demonstrations within prompts. This helps the NER model generate more robust pseudo-labels for the noisy training data.
  2. Stage II: Co-training with a Prompt-based Discriminator: A secondary discriminator model (based on RoBERTa) is trained on the clean guidance set. This discriminator uses specific prompts to evaluate noisy labels. For instance, it checks spans using prompts like [Sentence]. [Entity] is a [MASK] entity and categories using [Sentence]. [Entity] is a [MASK] [Category] entity. The discriminator's output recalibrates sample weights, prioritizing labels it trusts.
  3. Stage III: Fine-tuning: The final NER model is fine-tuned on the clean guidance set to further refine its accuracy.
The Self-Cleaning framework architecture, illustrating the two-stage co-training process between the NER model and the prompt-based discriminator.

Experimental Evaluation

The framework was tested on two types of noisy data: crowdsourced (CoNLL03-C) and distantly supervised (CoNLL03, OntoNotes 5.0, and Wikigold). The guidance sets were kept minimal, ranging from 50 to 500 sentences depending on the dataset scale.

Key Results

The Self-Cleaning method consistently outperformed state-of-the-art denoising baselines such as BOND, RoSTER, and NEEDLE. On the crowdsourced CoNLL03-C dataset, it achieved an F1 score of 0.862, a significant improvement over previous methods.

Performance comparison of Self-Cleaning against various baselines on the CoNLL03-C crowdsourced dataset.

Analysis revealed that the prompt-based discriminator is particularly effective at identifying span errors, which are often the most detrimental type of noise in NER tasks. Furthermore, the method demonstrated high sample efficiency; using only 200 clean instances, the framework achieved performance comparable to a standard model trained on 1,000 clean instances.

Conclusion

The Self-Cleaning framework provides a robust solution for training NER models on noisy data. By leveraging a small guidance set and a prompt-based discriminator, it effectively mitigates the impact of span and category errors. Future work may explore token-level discriminators and improved negative sampling techniques to further enhance denoising capabilities.

Summary updated

Abstract

To achieve state-of-the-art performance, one still needs to train NER models on large-scale, high-quality annotated data, an asset that is both costly and time-intensive to accumulate. In contrast, real-world applications often resort to massive low-quality labeled data through non-expert annotators via crowdsourcing and external knowledge bases via distant supervision as a cost-effective alternative. However, these annotation methods result in noisy labels, which in turn lead to a notable decline in performance. Hence, we propose to denoise the noisy NER data with guidance from a small set of clean instances. Along with the main NER model we train a discriminator model and use its outputs to recalibrate the sample weights. The discriminator is capable of detecting both span and category errors with different discriminative prompts. Results on public crowdsourcing and distant supervision datasets show that the proposed method can consistently improve performance with a small guidance set.
BibTeX
@inproceedings{chu2024improving,
  author = {Zhendong Chu and Ruiyi Zhang and Tong Yu and Rajiv Jain and Vlad I. Morariu and Jiuxiang Gu and Ani Nenkova},
  title = {{Improving a Named Entity Recognizer Trained on Noisy Data with a Few Clean Instances}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
  url = {https://openreview.net/forum?id=AIfmOITgFN},
}
Figure