Lexicon Infused Phrase Embeddings for Named Entity Resolution.

Alexandre Passos, Vineet Kumar, Andrew McCallum

TL;DR

This paper introduces a novel method for training word embeddings that leverages external lexicons to improve their relevance for Named Entity Recognition (NER). Their "lexicon-infused" embeddings achieve state-of-the-art results on the CoNLL 2003 NER task, outperforming previous methods that relied solely on unsupervised learning or less structured external knowledge.

What Problem Does This Paper Address?

Named Entity Recognition (NER) systems aim to identify and classify named entities (like people, organizations, and locations) in text. While traditional NER systems often rely on word clusters or lexicons for external knowledge, recent neural network-based language models (which produce word embeddings) have shown promise but historically underperformed. A key challenge is that these neural embeddings, while scalable, can be computationally expensive to train and their "nonlinear" nature makes it difficult to fully leverage their information within standard log-linear models like Conditional Random Fields (CRFs). Furthermore, existing methods for incorporating lexicons can be sensitive to noise. This paper seeks to bridge this gap by developing a more effective way to integrate lexicon information into word embeddings, making them more useful for NER.

Methods

The core contribution is a modification of the Skip-gram language model, a popular and efficient method for learning word embeddings. The authors propose "lexicon-infused Skip-gram models" that, in addition to predicting neighboring words, also predict whether a word belongs to specific lexicons. This injects task-specific supervisory signals directly into the embedding training process.

Key aspects of their method include:

  • Lexicon Integration: Instead of directly embedding lexicons into the word's binary tree structure (as in the standard Skip-gram), each lexicon is given its own binary classifier. This allows the model to learn embeddings that are influenced by lexicon membership without being overly sensitive to noisy lexicon entries.
  • Phrase Embeddings: The authors also extend their approach to learn embeddings for phrases, not just individual words, by considering phrases with high pointwise mutual information and those found in Wikipedia page titles.
  • NER System: They integrate these lexicon-infused phrase embeddings as features into a stacked linear-chain CRF model, a standard architecture for NER.

Datasets

The paper uses the following datasets:

  • CoNLL 2003: A widely used dataset for NER, with approximately 320k tokens for training, development, and testing. It includes four entity types: PERSON, LOCATION, ORGANIZATION, and MISCELLANEOUS.
  • Ontonotes 5.0: A larger and more challenging NER dataset with 1.6M tokens and eighteen entity types.
  • Wikipedia categories and US Census data: Used to construct the lexicons for training the lexicon-infused embeddings.
  • English Wikipedia and RCV1 Corpus: Used for learning phrase embeddings.

Key Insights and Ideas

  • Lexicons as Direct Supervision for Embeddings: Injecting lexicon membership as a training objective for word embeddings significantly improves their relevance for downstream tasks like NER, more so than simply using lexicons as features in a separate model.
  • Robustness to Noisy Lexicons: The proposed method of using separate classifiers for each lexicon makes the embedding training more robust to noisy or imperfect lexicon entries, a common problem with external knowledge sources.
  • Phrase Embeddings for NER: Learning embeddings for phrases, not just words, is crucial for achieving state-of-the-art NER performance, especially when these phrase embeddings are informed by external knowledge.
  • Efficiency of Lexicon-Infused Training: The lexicon-infused Skip-gram model can be trained on massive datasets (billions of tokens) in a single day, making it practical for large-scale applications.

Key Results

  • CoNLL 2003 NER: The lexicon-infused phrase embeddings achieved an F1 score of 90.90 on the CoNLL 2003 test set. This is a state-of-the-art result, matching a system that used massive private industrial query-log data, but is the first to achieve this using only publicly available data.
  • Ontonotes 5.0 NER: The lexicon-infused embeddings also provided better results than standard Skip-gram or Brown clusters on this dataset, though Skip-gram embeddings performed slightly better than lexicon-infused ones in this specific case.
  • Syntactic and Semantic Similarity Task: Lexicon-infused embeddings performed better than standard Skip-gram embeddings on a task designed to measure their ability to capture semantic and syntactic regularities.

Limitations and Future Work

  • Ontonotes Performance Nuance: While lexicon-infused embeddings generally improved performance, standard Skip-gram embeddings yielded marginally better results on the Ontonotes dataset. The authors note they did not analyze the test set errors for this specific case.
  • Future Work: The authors express interest in exploring the application of their lexicon-infused embeddings to other NLP tasks such as dependency parsing and coreference resolution. They also plan to investigate methods for improving embeddings by using error gradients directly from the NER task.

Summary updated

BibTeX
@inproceedings{passos2014lexicon,
  author = {Alexandre Passos and Vineet Kumar and Andrew McCallum},
  title = {{Lexicon Infused Phrase Embeddings for Named Entity Resolution.}},
  booktitle = {CoNLL},
  year = {2014},
  url = {https://openreview.net/forum?id=zFaaTpu1k},
}
Figure