Improving the Robustness of Distantly-Supervised Named Entity Recognition via Uncertainty-Aware Teacher Learning and Student-Student Collaborative Learning

Shuzheng Si, Helan Hu, Haozhe Zhao, Shuang Zeng, Kaikai An, Zefan Cai, Baobao Chang

TL;DR

Researchers have developed CENSOR, a new framework that significantly improves how AI models recognize named entities (like people or locations) when trained on noisy, automatically generated data. By combining "uncertainty-aware" filtering with a collaborative learning strategy between two "student" models, CENSOR avoids the common trap of overconfidently learning from incorrect labels, setting new performance records across five major benchmarks.

What Problem Does This Paper Address?

Named Entity Recognition (NER) is a foundational task in NLP, but training these models usually requires massive amounts of expensive, human-annotated data. To bypass this, researchers use Distant Supervision (DS-NER), where they automatically label text by matching it against existing knowledge bases (like Wikipedia).

However, as shown in Figure 1, this process is incredibly noisy:

  • Incomplete Annotation: If a name isn't in the database, it gets labeled as a "non-entity" (e.g., "Arafat" missed entirely).
  • Inaccurate Annotation: Context-free matching leads to wrong types (e.g., labeling "Washington" as a location when the text refers to a person).
Figure 1: A sample generated by DS-NER. “Amazon” and “Washington” are inaccurate annotations. “Arafat" and “rainforest" are the incomplete annotations.

Previous attempts to fix this used a "teacher-student" framework, where a teacher model generates "pseudo-labels" to train a student. The fatal flaw? Deep learning models are often poorly calibrated—they can be extremely confident even when they are dead wrong. This leads to "error propagation," where the student simply memorizes the teacher's mistakes.

Methods

The authors introduce CENSOR (unCertainty-aware tEacher aNd Student-student cOllaborative leaRning). The framework introduces two novel components to break the cycle of error propagation:

  1. Uncertainty-Aware Teacher Learning (UTL): Instead of trusting a teacher just because it is "confident," CENSOR measures uncertainty. It uses Monte Carlo Dropout—running the same text through the teacher multiple times with different neurons deactivated. If the predictions vary wildly, the label is deemed uncertain and masked out. Only high-confidence, low-uncertainty labels are used for training.
  2. Student-Student Collaborative Learning (SCL): Rather than one student following one teacher, CENSOR uses two parallel teacher-student pairs. These two students "talk" to each other. Specifically, they identify "small-loss" samples (data points the model finds easy and likely correct) and exchange these reliable labels. This peer-teaching prevents a student from becoming a carbon copy of its own potentially biased teacher.
Figure 2: General architecture of CENSOR, which consists of two teacher-student networks. [①] means the teacher network first generates pseudo labels. [②] means estimating the confidence and uncertainty of generated pseudo labels. [③] means selecting reliable pseudo labels according to confidence and uncertainty, where masked pseudo labels will not be used to update the student network. [④] means using Student-Student Collaborative Learning to transfer the reliable pseudo labels. [⑤] means using selected reliable pseudo labels to update the corresponding student network. [⑥] means updating a new teacher by shifting the weights of the trained student.

Datasets

The model was evaluated on five diverse DS-NER datasets to ensure broad applicability:

  • CoNLL03: The standard news-based benchmark.
  • OntoNotes 5.0: A large-scale multi-genre corpus.
  • Webpage: Noisy data from web crawls.
  • Wikigold: Wikipedia-based annotations.
  • Twitter: Highly informal and challenging social media text.

Key Insights and Ideas

  • Confidence $\neq$ Correctness: The paper highlights that in noisy environments, a model's "confidence score" is a biased indicator. Uncertainty (variance) is a much more robust signal for filtering out distant supervision errors.
  • The Power of Diversity: By using two different student networks, the system benefits from different "decision boundaries." One student might be better at recognizing patterns that the other misses, and the SCL mechanism allows this knowledge to be shared safely.
  • Full Data Exploration: Unlike previous methods that simply threw away "unreliable" samples, the collaborative learning component gives mislabeled samples a second chance to be correctly identified by the peer network, leading to better data utilization.

Key Results

CENSOR achieved state-of-the-art (SOTA) results on all five datasets, outperforming previous heavyweights like BOND, SCDL, and ATSEN.

  • Overall Performance: It improved the F1 score by an average of 1.87% over the previous best methods.
  • Robustness to Noise: As shown in Figure 3, CENSOR maintains high performance even when the "noise ratio" in the data reaches a staggering 80%. While other models' performance falls off a cliff, CENSOR's uncertainty filtering keeps it stable.
  • Significant Gains on Hard Data: On the notoriously difficult Twitter dataset, CENSOR improved the F1 score from 51.71 to 52.41.
Figure 3: F1 on CoNLL03 with different noise ratios.

Limitations and Future Work

The primary drawback of CENSOR is computational cost. Because the uncertainty estimation requires multiple forward passes (Monte Carlo Dropout), the self-training phase takes about four times longer than previous methods like ATSEN.

The authors also note that CENSOR is currently specialized for the DS-NER task. Future work will likely focus on optimizing the speed of uncertainty estimation and exploring whether this collaborative framework can be generalized to other noisy sequence-labeling tasks beyond named entity recognition.

Summary updated

Abstract

Distantly-Supervised Named Entity Recognition (DS-NER) is widely used in real-world scenarios. It can effectively alleviate the burden of annotation by matching entities in existing knowledge bases with snippets in the text but suffer from the label noise. Recent works attempt to adopt the teacher-student framework to gradually refine the training labels and improve the overall robustness. However, these teacher-student methods achieve limited performance because the poor calibration of the teacher network produces incorrectly pseudo-labeled samples, leading to error propagation. Therefore, we propose: (1) Uncertainty-Aware Teacher Learning that leverages the prediction uncertainty to reduce the number of incorrect pseudo labels in the self-training stage; (2) Student-Student Collaborative Learning that allows the transfer of reliable labels between two student networks instead of indiscriminately relying on all pseudo labels from its teacher, and further enables a full exploration of mislabeled samples rather than simply filtering unreliable pseudo-labeled samples. We evaluate our proposed method on five DS-NER datasets, demonstrating that our method is superior to the state-of-the-art DS-NER methods.
BibTeX
@inproceedings{si2024improving,
  author = {Shuzheng Si and Helan Hu and Haozhe Zhao and Shuang Zeng and Kaikai An and Zefan Cai and Baobao Chang},
  title = {{Improving the Robustness of Distantly-Supervised Named Entity Recognition via Uncertainty-Aware Teacher Learning and Student-Student Collaborative Learning}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
  doi = {10.18653/v1/2024.findings-acl.329},
}
Figure