Abstract
Neural network approaches to Named-Entity Recognition (NER) have significantly reduced the reliance on hand-crafted features. While some features persist in state-of-the-art systems, lexical features—with the exception of gazetteers—have been largely discarded. In this work, we demonstrate that lexical features remain highly valuable. We propose embedding words and entity types into a joint low-dimensional vector space trained via distant supervision on Wikipedia data. From this space, we pre-compute a Lexical Similarity (LS) feature vector for each word. When integrated into a vanilla recurrent neural network, this representation yields substantial improvements, establishing a new state-of-the-art F1 score of 87.95 on OntoNotes 5.0 and matching state-of-the-art performance on CoNLL-2003.
Introduction: Rethinking the Gazetteer
In the evolution of Named-Entity Recognition, neural networks have largely automated the feature engineering process. However, the gazetteer—a traditional list of known entities—remains a staple in many high-performing systems. Traditional gazetteers suffer from being binary (a word is either in the list or not), computationally expensive at test time, and focused solely on entity mentions rather than the surrounding context.
Ghaddar and Langlais argue that we haven't outgrown lexical features; rather, we need a more robust way to represent them. They introduce LS (Lexical Similarity) representations, which transform the discrete nature of gazetteers into dense, informative vectors.
The LS Representation
The core innovation is a 120-dimensional vector that encodes how similar any given word is to 120 different fine-grained entity types. The process involves three main steps:
- Distant Supervision: Using Wikipedia articles automatically annotated with 120 entity types (e.g.,
/person/director,/building/hotel). - Joint Embedding: Training word embeddings such that words and entity types share the same vector space.
- Similarity Mapping: Calculating the cosine similarity between a word and each of the 120 entity types to form the LS vector.
As shown in the table above, this method captures nuanced relationships. For instance, the word "directed" shows high similarity to the /person/director type, acting as a "contextual trigger" that helps the model anticipate upcoming entities.
Neural Architecture
The LS vectors are computed offline, making them highly efficient at inference time. These vectors are then fed into a standard Bi-LSTM-CRF architecture alongside traditional word and character embeddings.
The model combines four types of input features: word embeddings, character representations (captured via a nested Bi-LSTM), capitalization features, and the proposed LS features. This multi-faceted input allows the model to leverage both global semantic information and local lexical cues.
Key Insights and Results
The authors evaluated their approach on two major benchmarks: CoNLL-2003 and OntoNotes 5.0. The inclusion of LS features provided a significant performance boost:
- OntoNotes 5.0: Achieved a new state-of-the-art F1 score of 87.95, surpassing the previous best of 86.99.
- CoNLL-2003: Matched the state-of-the-art with an F1 of 91.73.
Beyond raw scores, the LS representation offers several advantages. It is robust to noise because it relies on embedding proximity rather than exact string matches. It also handles rare words effectively by leveraging their position in the joint embedding space. Furthermore, the ablation studies revealed that LS vectors and standard word embeddings capture complementary information, with their combination yielding the best results.
Conclusion
By reimagining gazetteers as dense similarity vectors, this work proves that lexical features are still vital for NER. The LS representation provides a computationally efficient, context-aware, and robust alternative to traditional lookup tables, pushing the boundaries of what "vanilla" neural architectures can achieve in sequence labeling tasks.
Summary updated