Abstract
Named Entity Recognition (NER) systems traditionally rely on semi-supervised information such as word clusters and curated lexicons. While neural word embeddings like Skip-gram offer scalable alternatives, they often lack the task-specific nuance required for high-performance NER. This paper introduces a novel approach to learning word and phrase embeddings by "infusing" domain-specific lexicons directly into the training objective. By modifying the Skip-gram model to predict both context and lexicon membership, we achieve state-of-the-art results. Our system reaches an F1 score of 90.90 on the CoNLL 2003 benchmark, matching the performance of systems trained on massive private datasets using only publicly available resources.
Introduction
Identifying and categorizing entities—such as people, locations, and organizations—is a cornerstone of natural language processing. Historically, state-of-the-art NER performance has been achieved through external knowledge sources: curated lexicons and word representations like Brown clustering. However, Brown clusters scale quadratically with vocabulary size, making them difficult to apply to massive modern corpora. Conversely, neural word embeddings scale efficiently but often fail to capture the specific semantic distinctions necessary for NER, as they are typically optimized for general semantic similarity.
Methodology: Lexicon-Infused Skip-gram
The core contribution of this work is the Lexicon-infused Skip-gram model. Building upon the standard Skip-gram architecture, which learns embeddings by predicting context words, we introduce a dual-objective training process. For every target word, the model is trained to predict:
- Surrounding Context: The standard unsupervised signal.
- Lexicon Membership: A supervised signal where each lexicon (e.g., US cities, surnames) acts as an auxiliary binary classifier.
This approach forces the embedding space to cluster words based on known entity types. To maintain efficiency and handle the sparsity of lexicon membership, the authors employ aggressive negative sub-sampling (utilizing only 1% of negative examples). This ensures the training remains balanced and scalable to billions of tokens.
Experimental Setup
The models were trained on a massive corpus combining English Wikipedia (8 million articles) and the RCV1 corpus (946,000 articles). Evaluation was conducted on two primary benchmarks:
- CoNLL 2003: A standard NER dataset with 4 entity types.
- Ontonotes 5.0: A more complex dataset featuring 18 entity types.
We utilized 22 lexicon classes derived from Wikipedia categories and US Census data, covering job titles, locations, and personal names.
Results and Discussion
The lexicon-infused embeddings significantly outperformed standard Skip-gram models and traditional clustering methods. On the CoNLL 2003 test set, the system achieved an F1 score of 90.90, matching the performance of systems that utilize private industrial query-log data. On Ontonotes 5.0, the system reached an F1 of 82.24.
Beyond NER, the infused embeddings also showed improved performance on general semantic similarity tasks, as shown in the table above. This suggests that task-specific supervision can enhance the overall quality of word representations.
Key Insights
- Contextual Disambiguation: Lexicons are often noisy (e.g., "He" as a surname vs. a pronoun). By training the model to predict lexicon membership in context, it learns to distinguish between entity and non-entity usages.
- Linear Compatibility: Injecting supervision into the embeddings makes them more informative for the linear-chain Conditional Random Fields (CRFs) typically used in NER pipelines.
- Dimensionality Paradox: The authors observed that 50-dimensional embeddings outperformed larger vectors for NER, suggesting that higher-dimensional embeddings may introduce complexity that simple CRFs cannot effectively leverage.
Conclusion
This research demonstrates that infusing lexicons into neural embeddings provides a scalable and effective way to incorporate domain knowledge into NLP models. Future work may explore applying this technique to other tasks such as dependency parsing and coreference resolution, as well as investigating end-to-end training where embeddings are updated based on the final task loss.
Summary updated