Context-aware Adversarial Training for Name Regularity Bias in Named Entity Recognition

Abbas Ghaddar, Philippe Langlais, Ahmad Rashid, Mehdi Rezagholizadeh

Abstract: In this work, we examine the ability of NER models to use contextual information when predicting the type of an ambiguous entity. We introduce NRB, a new testbed carefully designed to diagnose Name Regularity Bias of NER models. Our results indicate that all state-of-the-art models we tested show such a bias; BERT fine-tuned models significantly outperforming feature-based (LSTM-CRF) ones on NRB, despite having comparable (sometimes lower) performance on standard benchmarks. To mitigate this bias, we propose a novel model-agnostic training method that adds learnable adversarial noise to some entity mentions, thus enforcing models to focus more strongly on the contextual signal, leading to significant gains on NRB. Combining it with two other training strategies, data augmentation and parameter freezing, leads to further gains.

Introduction: The Challenge of Name Regularity Bias

Modern NLP models are notorious for exploiting surface-level patterns—dataset biases—rather than learning the underlying task. In Named Entity Recognition (NER), this manifests as Name Regularity Bias. If a model frequently sees "Washington" as a Location in its training data, it may blindly predict Location even in a context like "Washington vetoed the bill," where it clearly refers to a Person.

Previous attempts to diagnose this bias involved artificially swapping entities in test sets (e.g., replacing "Paris" with "John"). However, this creates unnatural sentences that can confuse models for the wrong reasons. This research addresses the bias using genuine, natural language and proposes a training methodology to overcome it without requiring massive amounts of new, specialized data.

Methods

To force models to prioritize contextual clues over entity memorization, the authors introduced three key training strategies:

  1. Entity Masking: Inspired by BERT's pre-training, this data augmentation technique replaces some entities in the training data with a [MASK] token. If the entity is missing, the model must use the surrounding words to guess its type.
  2. Parameter Freezing: The authors freeze the bottom half of BERT (including embeddings) during fine-tuning. This prevents the model from "unlearning" the general contextual representations it acquired during its initial pre-training phase.
  3. Adversarial Noise (adv): This core algorithmic novelty involves adding a learnable noise vector to a token's input representation. By simulating a "noisy" or incorrect entity type (e.g., adding a Person noise vector to a Location entity), the system makes the entity's name an unreliable signal. To minimize loss, the network is forced to look outward at the surrounding context.
Overview of the context-aware adversarial training method.

Datasets

The study evaluates models on standard benchmarks (CoNLL-2003, OntoNotes 5.0) and introduces two critical diagnostic sets:

  • NRB (Name Regularity Bias): A new dataset of 5,275 natural English sentences mined from Wikipedia disambiguation pages. These contain highly ambiguous entities (e.g., "Gonzales" used as a city rather than a person).
  • WTS (Witness): A control set containing the exact same entities as NRB, but used in their most common, easy-to-guess contexts.

Key Insights

  • Bias vs. Domain Shift: Models perform poorly on the NRB dataset but excellently on the WTS control set. Because both datasets use the same entities from the same domain (Wikipedia), the failure on NRB definitively proves models suffer from name regularity bias rather than a lack of vocabulary knowledge.
  • BERT's Advantage: Fine-tuned BERT models significantly outperform feature-based models (like LSTM-CRFs) on ambiguous entities. This is largely because BERT's Masked Language Modeling (MLM) pre-training inherently teaches it to rely on context.
  • Attention Shift: Analysis of BERT's attention heads reveals that adversarial training physically changes how the model processes data. In the upper layers, significantly fewer attention heads point to the entity itself, proving the model is actively looking at surrounding context tokens.

Key Results

Standard feature-based models (LSTM-CRF) show a massive performance drop, scoring as low as 27-38% F1 on the NRB dataset despite scoring >90% on standard test sets. While standard fine-tuned BERT-large performs better at 75.6% F1, it still shows a significant gap compared to standard benchmarks.

Performance comparison of various models on standard benchmarks vs. the NRB diagnostic set.

Combining all three mitigation strategies (Masking + Freezing + Adversarial Noise) pushed BERT-large's NRB score to 89.7% F1 (a ~14 point jump) while maintaining a stellar 92.8% on the standard CoNLL test set. This robustness holds even in low-resource settings and across multiple languages.

Limitations and Future Work

While the adversarial training method yielded drastic improvements, the multilingual evaluation relied on machine translation, which resulted in discarding up to 53% of sentences due to quality issues. Future work includes building native NRB corpora for other languages and expanding the benchmark to evaluate multi-word entities and non-sequential NER architectures.

Summary updated

Abstract

In this work, we examine the ability of NER models to use contextual information when predicting the type of an ambiguous entity. We introduce NRB, a new testbed carefully designed to diagnose Name Regularity Bias of NER models. Our results indicate that all state-of-the-art models we tested show such a bias; BERT fine-tuned models significantly outperforming feature-based (LSTM-CRF) ones on NRB, despite having comparable (sometimes lower) performance on standard benchmarks. To mitigate this bias, we propose a novel model-agnostic training method that adds learnable adversarial noise to some entity mentions, thus enforcing models to focus more strongly on the contextual signal, leading to significant gains on NRB. Combining it with two other training strategies, data augmentation and parameter freezing, leads to further gains.
BibTeX
@article{ghaddar2021context,
  author = {Abbas Ghaddar and Philippe Langlais and Ahmad Rashid and Mehdi Rezagholizadeh},
  title = {{Context-aware Adversarial Training for Name Regularity Bias in Named Entity Recognition}},
  journal = {Trans. Assoc. Comput. Linguistics},
  year = {2021},
  url = {https://openreview.net/forum?id=Bmxql7oWjAG},
}
Figure