Introduction
Named Entity Recognition (NER) is a fundamental task in natural language processing, essential for identifying entities like names, locations, and organizations. Despite the high performance of modern Transformer-based models like BERT, they often fail in production environments when encountering new entities or ambiguous contexts. For instance, a model might miss a newly launched product name or misclassify a location as a sports team.
The traditional solution to these errors is a slow, resource-intensive cycle: annotating the "bad cases," merging them into the training set, retraining the entire model, and redeploying. This paper introduces TyBERT (Type Enhanced BERT), a framework designed to make NER models "steerable." By integrating an external gazetteer, developers can "hot-fix" errors in real-time simply by updating a lookup table, bypassing the need for retraining.
Methodology
TyBERT enhances the standard BERT architecture through two primary components:
- Word-Type Pair Sequences: The model matches input text against a gazetteer. If a word matches multiple entries (e.g., "London" as both a location and a song title), all potential types are passed to the model alongside the word embeddings.
- Type Adapter (TA): A lightweight adapter layer is inserted after the first Transformer layer. This adapter employs a two-level cross-attention mechanism. First, it computes attention between BERT's hidden states and the embeddings of potential entity types from the gazetteer. Second, it refines the hidden states using this type-aware information.
To prevent the model from becoming overly reliant on the gazetteer, the authors introduce a noise rate parameter ($\lambda$) during training. By intentionally injecting incorrect types or random spans into the gazetteer, the model is forced to balance external lookup information with linguistic context. A noise rate of $\lambda=0.05$ was found to be the optimal balance.
Experimental Setup
The effectiveness of TyBERT was evaluated across four datasets in two languages:
- English: OntoNotes v5.0 (18 entity types) and Few-NERD (66 fine-grained types).
- Chinese: OntoNotes v4.0 (8 types) and Weibo (4 types from social media).
Results and Discussion
TyBERT consistently outperformed standard BERT and simple rule-based ensembles across all benchmarks. The most significant gains were observed in fine-grained and noisy datasets.
As shown in the results, TyBERT achieved a massive F1-score improvement on the Few-NERD dataset, jumping from 68.4 (BERT) to 87.31. On OntoNotes v5.0, it reached an F1 of 94.74.
Beyond static benchmarks, the authors demonstrated the "steerability" of the model. By updating the gazetteer with errors identified in the test set, performance improved significantly without modifying model weights. For example, in Chinese OntoNotes, this "hot-fix" approach boosted the F1-score from 78.43 to 85.97.
Conclusion
TyBERT provides a practical solution for maintaining NER models in production. By decoupling entity knowledge from the model weights via a gazetteer and a specialized adapter, it allows for rapid error correction. Future work will focus on expanding gazetteer coverage using large-scale knowledge bases like Wikipedia and improving the model's ability to resolve complex contextual ambiguities when multiple types are suggested by the gazetteer.
Summary updated