An Attentive Fine-Grained Entity Typing Model with Latent Type Representation

Ying Lin, Heng JiEMNLP 2019 · OpenReview

Abstract: Ying Lin, Heng Ji. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP).

2019.

Introduction

Fine-grained entity typing (FET) is the task of assigning specific, hierarchical labels to entity mentions based on their context. For instance, in the sentence "Rogers is in the game," a robust FET system should identify "Rogers" not merely as a Person, but specifically as an Athlete. Researchers from UIUC have introduced a new state-of-the-art model that significantly improves FET performance by moving beyond treating labels as independent categories and utilizing sophisticated attention mechanisms.

Addressing the Gaps in Entity Typing

The authors identify three primary limitations in previous FET research:

  1. Label Independence: Most models treat labels like Artist, Actor, and Person as independent binary classification problems, ignoring the inherent hierarchical correlations (e.g., an Artist is a subclass of Person).
  2. Static Embeddings: Reliance on fixed word embeddings (like GloVe) prevents models from capturing polysemy or context-specific nuances.
  3. Coarse Attention: Existing attention mechanisms often focus solely on the context surrounding a mention, neglecting informative words within the mention itself (e.g., "Department" in "UW Department of Chemistry").

The Proposed Framework

The proposed model addresses these gaps through a three-component architecture:

  • Contextualized Encoding: The system utilizes ELMo (and BERT) to generate dynamic word representations that change based on the surrounding sentence.
  • Two-Step Mention-Aware Attention: The model first applies attention to words inside the entity mention to identify the "head" word. It then uses this mention representation to query the surrounding context, applying a relative position term to prioritize nearby words.
  • Hybrid Classification Model: This novel component uses two parallel classifiers. A Binary Relevance classifier predicts types independently, while a Latent Type Representation classifier (inspired by Principle Label Space Transformation) predicts a low-dimensional vector capturing underlying type correlations.
The proposed model architecture featuring two-step attention and a hybrid classification layer.

Experimental Evaluation

The model was benchmarked across four diverse datasets:

  • OntoNotes: 87 types across a 3-layer hierarchy.
  • FIGER: 113 types across a 2-layer hierarchy (Wikipedia-derived).
  • KNET: 74 types across a 2-layer hierarchy.
  • BBN: 56 types across a 2-layer hierarchy (Wall Street Journal-derived).

Key Results and Ablation Analysis

The model achieved substantial gains over previous state-of-the-art (SOTA) methods. On OntoNotes, it reached 63.8% Strict Accuracy and 82.9% Macro F1, a significant jump from the previous SOTA of 59.5% and 76.8%, respectively. On FIGER, it achieved an 83.0% Macro F1, representing a 4% absolute gain.

An ablation study confirmed that each architectural choice—contextualized embeddings, latent representations, and the two-step attention mechanism—contributed significantly to the overall performance.

Ablation study results showing the performance contribution of ELMo, latent representation, and two-step attention.

Conclusion and Future Directions

By treating labels as a correlated space rather than isolated categories and implementing mention-aware attention, this model sets a new standard for fine-grained entity typing. While the model excels at general types, the authors note that performance on extremely fine-grained "long tail" types remains a challenge due to data sparsity. Future work will focus on improving performance for these rare types and integrating the model into broader NLP pipelines for tasks like relation and event extraction.

Summary updated

BibTeX
@inproceedings{lin2019attentive,
  author = {Ying Lin and Heng Ji},
  title = {{An Attentive Fine-Grained Entity Typing Model with Latent Type Representation}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2019},
  url = {https://openreview.net/forum?id=tSFkk4agqnv},
}
Figure