Robust Lexical Features for Improved Neural Network Named-Entity Recognition

Abbas Ghaddar, Philippe Langlais

Abstract

Neural network approaches to Named-Entity Recognition (NER) have significantly reduced the reliance on hand-crafted features. While some features persist in state-of-the-art systems, lexical features—with the exception of gazetteers—have been largely discarded. In this work, we demonstrate that lexical features remain highly valuable. We propose embedding words and entity types into a joint low-dimensional vector space trained via distant supervision on Wikipedia data. From this space, we pre-compute a Lexical Similarity (LS) feature vector for each word. When integrated into a vanilla recurrent neural network, this representation yields substantial improvements, establishing a new state-of-the-art F1 score of 87.95 on OntoNotes 5.0 and matching state-of-the-art performance on CoNLL-2003.

Introduction: Rethinking the Gazetteer

In the evolution of Named-Entity Recognition, neural networks have largely automated the feature engineering process. However, the gazetteer—a traditional list of known entities—remains a staple in many high-performing systems. Traditional gazetteers suffer from being binary (a word is either in the list or not), computationally expensive at test time, and focused solely on entity mentions rather than the surrounding context.

Ghaddar and Langlais argue that we haven't outgrown lexical features; rather, we need a more robust way to represent them. They introduce LS (Lexical Similarity) representations, which transform the discrete nature of gazetteers into dense, informative vectors.

The LS Representation

The core innovation is a 120-dimensional vector that encodes how similar any given word is to 120 different fine-grained entity types. The process involves three main steps:

  1. Distant Supervision: Using Wikipedia articles automatically annotated with 120 entity types (e.g., /person/director, /building/hotel).
  2. Joint Embedding: Training word embeddings such that words and entity types share the same vector space.
  3. Similarity Mapping: Calculating the cosine similarity between a word and each of the 120 entity types to form the LS vector.
Examples of words and their similarity scores to various entity types.

As shown in the table above, this method captures nuanced relationships. For instance, the word "directed" shows high similarity to the /person/director type, acting as a "contextual trigger" that helps the model anticipate upcoming entities.

Neural Architecture

The LS vectors are computed offline, making them highly efficient at inference time. These vectors are then fed into a standard Bi-LSTM-CRF architecture alongside traditional word and character embeddings.

The Bi-LSTM-CRF architecture integrating LS features alongside standard embeddings.

The model combines four types of input features: word embeddings, character representations (captured via a nested Bi-LSTM), capitalization features, and the proposed LS features. This multi-faceted input allows the model to leverage both global semantic information and local lexical cues.

Key Insights and Results

The authors evaluated their approach on two major benchmarks: CoNLL-2003 and OntoNotes 5.0. The inclusion of LS features provided a significant performance boost:

  • OntoNotes 5.0: Achieved a new state-of-the-art F1 score of 87.95, surpassing the previous best of 86.99.
  • CoNLL-2003: Matched the state-of-the-art with an F1 of 91.73.

Beyond raw scores, the LS representation offers several advantages. It is robust to noise because it relies on embedding proximity rather than exact string matches. It also handles rare words effectively by leveraging their position in the joint embedding space. Furthermore, the ablation studies revealed that LS vectors and standard word embeddings capture complementary information, with their combination yielding the best results.

Conclusion

By reimagining gazetteers as dense similarity vectors, this work proves that lexical features are still vital for NER. The LS representation provides a computationally efficient, context-aware, and robust alternative to traditional lookup tables, pushing the boundaries of what "vanilla" neural architectures can achieve in sequence labeling tasks.

Summary updated

Abstract

Neural network approaches to Named-Entity Recognition reduce the need for carefully hand-crafted features. While some features do remain in state-of-the-art systems, lexical features have been mostly discarded, with the exception of gazetteers. In this work, we show that this is unfair: lexical features are actually quite useful. We propose to embed words and entity types into a low-dimensional vector space we train from annotated data produced by distant supervision thanks to Wikipedia. From this, we compute - offline - a feature vector representing each word. When used with a vanilla recurrent neural network model, this representation yields substantial improvements. We establish a new state-of-the-art F1 score of 87.95 on ONTONOTES 5.0, while matching state-of-the-art performance with a F1 score of 91.73 on the over-studied CONLL-2003 dataset.
BibTeX
@inproceedings{ghaddar2018robust,
  author = {Abbas Ghaddar and Philippe Langlais},
  title = {{Robust Lexical Features for Improved Neural Network Named-Entity Recognition}},
  booktitle = {International Conference on Computational Linguistics},
  year = {2018},
  url = {https://openreview.net/forum?id=6ZgnajgT6ja},
}
Figure