Dependency-Guided LSTM-CRF for Named Entity Recognition

Zhanming Jie, Wei LuEMNLP 2019 · OpenReview

TL;DR

This paper introduces a novel "dependency-guided" LSTM-CRF model that leverages the rich structural information present in dependency parse trees to significantly improve Named Entity Recognition (NER). By explicitly encoding the relationships between words as defined by their syntactic dependencies, the model achieves state-of-the-art performance, particularly for longer and more complex named entities.

What Problem Does This Paper Address?

Named Entity Recognition (NER) is a fundamental task in Natural Language Processing (NLP) that aims to identify and classify named entities (like persons, organizations, locations) within text. While traditional NER models often rely on sequential information, they struggle to capture long-distance dependencies and complex syntactic relationships between words. Dependency trees, which represent the grammatical structure of a sentence, offer a wealth of information about these relationships. However, effectively integrating this structural information into NER models, especially to capture non-local interactions, has remained a research challenge. This paper addresses this gap by proposing a method to fully utilize complete dependency tree structures for improved NER.

Methods

The core innovation is a Dependency-Guided LSTM-CRF (DGLSTM-CRF) model. This model builds upon the standard BiLSTM-CRF architecture, which is a strong baseline for sequence labeling tasks like NER. The key enhancements are:

  • Dependency-Encoded Input Representations: Instead of just using word embeddings, the input representation for each word is augmented with the embedding of its parent word in the dependency tree and the embedding of the dependency relation connecting them. This directly injects syntactic information at the input layer.
  • Dependency Tree Propagation: The model uses multiple layers of BiLSTMs. Crucially, the hidden states are not only propagated sequentially but also along the dependency tree structure. An "interaction function" (e.g., MLP, concatenation) is used to combine the hidden states of a child node with its parent node's hidden state. This allows information to flow through the dependency tree, capturing longer-range interactions.
  • Graph Convolutional Network (GCN) Analogy: The interaction mechanism, particularly when using an MLP with ReLU activation, is analogous to a Graph Convolutional Network (GCN), enabling the model to reason over the graph structure of the dependency tree.

Datasets

The authors evaluated their model on several standard datasets:

  • OntoNotes 5.0 (English and Chinese): Large-scale datasets containing both constituency and named entity annotations.
  • SemEval-2010 Task 1 (Catalan and Spanish): Datasets originally designed for coreference resolution but also containing dependency and named entity annotations.
  • CoNLL-2003 (English): A widely used benchmark for NER.

Dependency trees for these datasets were generated using rule-based tools or predicted by a dependency parser.

Key Insights and Ideas

  • Syntactic Structure is Crucial for NER: The paper strongly demonstrates that explicitly modeling syntactic relationships via dependency trees significantly boosts NER performance, especially for longer entities.
  • Entities Often Form Subtrees: A key observation is that many named entities form complete subtrees within the larger dependency parse tree. This structural property makes dependency-guided models particularly effective.
  • Grandchild Dependencies Matter: The model's ability to capture "grandchild dependencies" (i.e., relationships beyond immediate parent-child) is shown to be beneficial for recognizing complex entities.
  • Dependency Quality is Key: The performance gains are substantial when using high-quality, gold-standard dependency parses. While the model still outperforms baselines with predicted dependencies, the gap highlights the importance of accurate parsing.

Key Results

  • State-of-the-Art Performance: The DGLSTM-CRF model achieved state-of-the-art results on several datasets, outperforming strong baselines like BiLSTM-CRF and BiLSTM-GCN-CRF.
  • Significant Improvement for Long Entities: The model showed particularly strong improvements for entities with lengths greater than 2, with analysis indicating that long-distance and grandchild dependencies play a crucial role.
  • Robustness Across Languages: The effectiveness of the dependency-guided approach was demonstrated across English, Chinese, Catalan, and Spanish.
  • ELMo Integration: The model's performance was further enhanced when combined with contextualized word embeddings like ELMo, showing that dependency information complements other advanced representations.

Limitations and Future Work

  • Dependency Quality Dependence: The paper acknowledges that the model's performance is sensitive to the quality of the dependency parses. Using predicted dependencies, while still beneficial, leads to a performance drop compared to gold dependencies.
  • Joint Modeling: The authors suggest that future work could explore joint NER and dependency parsing models, treating entities as integral parts of the dependency tree structure.
  • Deeper GCN Layers: While stacking more GCN layers didn't yield improvements in their experiments, further exploration of deeper graph-based reasoning might be beneficial.

Summary updated

Abstract

Dependency tree structures capture long-distance and syntactic relationships between words in a sentence. The syntactic relations (e.g., nominal subject, object) can potentially infer the existence of certain named entities. In addition, the performance of a named entity recognizer could benefit from the long-distance dependencies between the words in dependency trees. In this work, we propose a simple yet effective dependency-guided LSTM-CRF model to encode the complete dependency trees and capture the above properties for the task of named entity recognition (NER). The data statistics show strong correlations between the entity types and dependency relations. We conduct extensive experiments on several standard datasets and demonstrate the effectiveness of the proposed model in improving NER and achieving state-of-the-art performance. Our analysis reveals that the significant improvements mainly result from the dependency relations and long-distance interactions provided by dependency trees.
BibTeX
@inproceedings{jie2019dependency,
  author = {Zhanming Jie and Wei Lu},
  title = {{Dependency-Guided LSTM-CRF for Named Entity Recognition}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2019},
  url = {https://openreview.net/forum?id=A-HFp-6nUHy},
}
Figure