Robust Lexical Features for Improved Neural Network Named-Entity Recognition

Abbas Ghaddar, Philippe Langlais

Abstract

Modern Named-Entity Recognition (NER) systems have largely transitioned from manual feature engineering to deep neural networks. While effective, these models often overlook the rich information contained in traditional lexical features or rely on computationally expensive gazetteers. This paper introduces LS (Lexical Similarity) representations, a compact 120-dimensional feature vector that captures the similarity between words and entity types. By embedding words and fine-grained entity types into a joint vector space, the authors achieved a new state-of-the-art F1 score on the OntoNotes 5.0 dataset while maintaining high inference efficiency.

The Limitations of Traditional Gazetteers

Named-Entity Recognition involves identifying and categorizing entities like people, places, and organizations. While neural NER systems are powerful, they often struggle with rare words or lack the domain-specific knowledge found in gazetteers (predefined entity lists). However, standard gazetteers suffer from three major flaws:

  1. Binary Representation: They provide a simple "yes/no" membership check, ignoring the probability that a word might belong to multiple categories depending on context.
  2. Computational Overhead: Matching every n-gram in a sentence against massive lexicons is expensive at test time.
  3. Non-entity Blindness: Gazetteers provide no signal for "trigger words" (e.g., "born", "directed") that are not entities themselves but strongly suggest an entity is nearby.

Methods: LS Representations and Model Architecture

The authors propose a Bi-LSTM-CRF model enhanced by a novel LS representation. The core innovation is a joint vector space where both words and 120 fine-grained entity types (from the WiFiNE resource) reside.

To learn these embeddings, the authors used FastText on a 1.3 billion token corpus. They created two versions of every sentence: the original text and a version where entity mentions were replaced by their type labels. This allows the model to learn proximity between words and the entity types they frequently associate with. For any given word, the LS vector is computed as the cosine similarity between that word’s embedding and each of the 120 entity type embeddings.

The Bi-LSTM-CRF architecture incorporating Word, Char, Caps, and the proposed LS (Lexical Similarity) embeddings.

Key Insights: Soft Preferences and Contextual Triggers

Unlike a binary gazetteer, the LS vector provides "soft preferences." For example, the word "Hilton" might show high similarity to both /building/hotel and /person/actor, allowing the neural network to resolve the ambiguity using sentence context.

Furthermore, the method captures "contextual triggers." Verbs like "directed" or "won" show high similarity to types like /person/director or /award, providing the model with a strong prior even when the entity itself is rare or unseen.

Examples of cosine similarity (Sim) between words and fine-grained entity types, illustrating how LS features capture both entity membership and contextual triggers.

Results and Performance

The model was evaluated on two major benchmarks:

  • OntoNotes 5.0: Achieved a new state-of-the-art F1 score of 87.95, outperforming previous models that used more complex architectures.
  • CoNLL-2003: Reached an F1 score of 91.73, matching systems that rely on much heavier language model embeddings.

Because LS features are pre-computed offline, the system maintains high efficiency during inference, avoiding the real-time matching overhead associated with traditional large-scale gazetteers.

Conclusion

LS representations offer a robust, efficient way to re-integrate lexical knowledge into neural NER systems. By moving from binary flags to continuous similarity scores in a joint embedding space, the model gains a nuanced understanding of both entities and the linguistic contexts that signal them. Future work may explore how these static features interact with modern contextualized embeddings like BERT.

Summary updated

Abstract

Abbas Ghaddar, Phillippe Langlais. Proceedings of the 27th International Conference on Computational Linguistics. 2018.
BibTeX
@inproceedings{ghaddar2018robust,
  author = {Abbas Ghaddar and Philippe Langlais},
  title = {{Robust Lexical Features for Improved Neural Network Named-Entity Recognition}},
  booktitle = {International Conference on Computational Linguistics},
  year = {2018},
  url = {https://openreview.net/forum?id=6ZgnajgT6ja},
}
Figure