Abstract
Standard name tagging models often struggle with rare or unseen entities because they treat all word embeddings as equally reliable. This paper introduces a reliability-aware gating mechanism that uses word frequency signals to dynamically balance word embeddings with character-level and contextual features. Evaluated on the OntoNotes 5.0 corpus, the proposed model shows significant performance gains, particularly in cross-genre scenarios where entity overlap is minimal.
Introduction
Name tagging, a fundamental task in Named Entity Recognition (NER), involves identifying and classifying entities such as people, organizations, and locations. Modern neural models rely heavily on pre-trained word embeddings (e.g., GloVe). However, these embeddings are subject to the "long tail" distribution: while common words have high-quality vectors, rare words—which names often are—result in noisy and unreliable representations. Current models are typically "reliability-agnostic," consuming all vectors with equal weight regardless of their quality.
Methodology: Reliability-Aware Gating
The authors propose a model that utilizes Reliability Signals to guide a dynamic feature composition process at two distinct levels. These signals are derived from word frequencies in both the pre-training corpus and the task-specific training set, serving as a proxy for embedding trustworthiness.
Word Representation Level Gating
Instead of simple concatenation, the model uses a gate to weigh word embeddings against character-level representations. If a word is rare (e.g., "MedChem"), the gate suppresses the noisy word vector and prioritizes character-level clues like prefixes or suffixes. This is further enhanced by splitting the character representation into semantic and morphological components.
Feature Extraction Level Gating
A key innovation is the introduction of "context-only" features—representations derived strictly from surrounding words, excluding the target word itself. A second set of gates balances the standard LSTM hidden state with these context-only features. This forces the model to leverage linguistic environments (e.g., "a small town like...") when the target word is unknown.
Experimental Results
The model was evaluated on the OntoNotes 5.0 corpus across six diverse genres, including Newswire, Telephone Conversations, and Weblogs.
- In-Genre Gains: The model outperformed the state-of-the-art LSTM-CNN-CRF baseline across all genres, with F-score improvements of up to 6.2% in the Telephone Conversation genre.
- Cross-Genre Robustness: In cross-genre experiments (training on one genre and testing on another), the model achieved an average F-score gain of 2.3%, with improvements as high as 9.6% in the most challenging cases.
- Handling Unseen Entities: Qualitative analysis demonstrated that the model effectively identifies names it has never seen by correctly identifying when context is more reliable than an unknown word's vector.
Conclusion
By making neural models "aware" of embedding reliability through frequency-based signals, this work provides a robust fallback mechanism for handling the long tail of language. While the current implementation uses LSTMs, the gating logic is complementary to modern Transformer-based architectures. Future research may explore more sophisticated reliability signals and apply this dynamic composition to tasks like event extraction.
Summary updated