Type Enhanced BERT for Correcting NER Errors

Kuai Li, Chen Chen, Tao Yang, Tianming Du, Peijie Yu, Dong Du, Feng ZhangACL 2023 · DOI

Introduction

Named Entity Recognition (NER) is a fundamental task in natural language processing, essential for identifying entities like names, locations, and organizations. Despite the high performance of modern Transformer-based models like BERT, they often fail in production environments when encountering new entities or ambiguous contexts. For instance, a model might miss a newly launched product name or misclassify a location as a sports team.

The traditional solution to these errors is a slow, resource-intensive cycle: annotating the "bad cases," merging them into the training set, retraining the entire model, and redeploying. This paper introduces TyBERT (Type Enhanced BERT), a framework designed to make NER models "steerable." By integrating an external gazetteer, developers can "hot-fix" errors in real-time simply by updating a lookup table, bypassing the need for retraining.

Workflow of TyBERT: Correcting NER errors by updating the gazetteer without retraining.

Methodology

TyBERT enhances the standard BERT architecture through two primary components:

  1. Word-Type Pair Sequences: The model matches input text against a gazetteer. If a word matches multiple entries (e.g., "London" as both a location and a song title), all potential types are passed to the model alongside the word embeddings.
  2. Type Adapter (TA): A lightweight adapter layer is inserted after the first Transformer layer. This adapter employs a two-level cross-attention mechanism. First, it computes attention between BERT's hidden states and the embeddings of potential entity types from the gazetteer. Second, it refines the hidden states using this type-aware information.

To prevent the model from becoming overly reliant on the gazetteer, the authors introduce a noise rate parameter ($\lambda$) during training. By intentionally injecting incorrect types or random spans into the gazetteer, the model is forced to balance external lookup information with linguistic context. A noise rate of $\lambda=0.05$ was found to be the optimal balance.

Experimental Setup

The effectiveness of TyBERT was evaluated across four datasets in two languages:

  • English: OntoNotes v5.0 (18 entity types) and Few-NERD (66 fine-grained types).
  • Chinese: OntoNotes v4.0 (8 types) and Weibo (4 types from social media).

Results and Discussion

TyBERT consistently outperformed standard BERT and simple rule-based ensembles across all benchmarks. The most significant gains were observed in fine-grained and noisy datasets.

Performance comparison of TyBERT against BERT baselines across English and Chinese datasets.

As shown in the results, TyBERT achieved a massive F1-score improvement on the Few-NERD dataset, jumping from 68.4 (BERT) to 87.31. On OntoNotes v5.0, it reached an F1 of 94.74.

Beyond static benchmarks, the authors demonstrated the "steerability" of the model. By updating the gazetteer with errors identified in the test set, performance improved significantly without modifying model weights. For example, in Chinese OntoNotes, this "hot-fix" approach boosted the F1-score from 78.43 to 85.97.

Conclusion

TyBERT provides a practical solution for maintaining NER models in production. By decoupling entity knowledge from the model weights via a gazetteer and a specialized adapter, it allows for rapid error correction. Future work will focus on expanding gazetteer coverage using large-scale knowledge bases like Wikipedia and improving the model's ability to resolve complex contextual ambiguities when multiple types are suggested by the gazetteer.

Summary updated

BibTeX
@inproceedings{li2023type,
  author = {Kuai Li and Chen Chen and Tao Yang and Tianming Du and Peijie Yu and Dong Du and Feng Zhang},
  title = {{Type Enhanced BERT for Correcting NER Errors}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
  doi = {10.18653/v1/2023.findings-acl.445},
}
Figure