Semantic Relation Classification via Bidirectional LSTM Networks with Entity-aware Attention using Latent Entity Typing

Joohong Lee, Sangwoo Seo, Yong Suk Choi

Abstract

Classifying semantic relations between entity pairs in sentences is a fundamental task in Natural Language Processing (NLP). Traditional models often depend on external linguistic tools like dependency parsers or POS taggers, which can introduce error propagation and increase computational overhead. We propose a novel end-to-end recurrent neural model that integrates an entity-aware attention mechanism with a Latent Entity Typing (LET) method. Our approach effectively utilizes entity information and latent types as features, achieving state-of-the-art performance on the SemEval-2010 Task 8 dataset without requiring high-level features.

Introduction

Relation Classification (RC) aims to identify the semantic relationship between two entities within a sentence (e.g., identifying "Cause-Effect" in "The crash was caused by the attack"). Historically, this task relied heavily on features extracted from external NLP toolkits. However, these tools are often slow and their errors can negatively impact the final classification.

While deep learning has moved toward end-to-end models, many existing architectures fail to prioritize the entities themselves—the most informative parts of the sentence for this task. This research addresses these gaps by focusing on entity-aware attention and automated entity categorization.

Methods

The proposed model consists of four primary layers designed to process raw text into a relational classification:

  1. Word Representation: Input words are mapped to dense vectors using pre-trained GloVe embeddings.
  2. Self-Attention: A multi-head attention mechanism captures global dependencies between words, providing contextualized representations before sequential processing.
  3. Bidirectional LSTM (BiLSTM): This layer processes the sequence in both forward and backward directions to capture long-range dependencies.
  4. Entity-aware Attention with Latent Entity Typing (LET): This core innovation calculates attention weights by considering the relative position of words to entities and their "latent types." Instead of using fixed categories (like "Person"), LET learns to represent entities as a mixture of $K$ learned type vectors.
The proposed model architecture featuring BiLSTM, Self-Attention, and Entity-aware Attention with Latent Entity Typing (LET).

Key Insights and Results

The model was evaluated on the SemEval-2010 Task 8 dataset, which contains 10,717 sentences across 19 possible relation labels.

Performance and Efficiency

The model achieved a state-of-the-art 85.2% F1-score, surpassing models that utilize complex dependency features. Ablation studies confirmed that the inclusion of Latent Entity Typing (LET) provided a significant performance boost (0.5%), demonstrating that "soft" semantic clustering helps the model generalize better than rigid, external tags.

Latent Entity Typing (LET)

A key finding is that the model's learned latent types correspond to meaningful semantic clusters. For instance, entities were automatically grouped into categories such as machinery, disasters, or professional roles without explicit supervision.

t-SNE visualization of the Latent Entity Typing (LET) results, showing the model's ability to cluster entities into semantic categories.

Conclusion

By focusing on entity-aware attention and latent typing, this model removes the dependency on error-prone external NLP tools while improving classification accuracy. Future work may involve dynamically determining the number of latent types and testing the architecture on more specialized domains like medical or legal texts.

Summary updated

Abstract

Classifying semantic relations between entity pairs in sentences is an important task in Natural Language Processing (NLP). Most previous models for relation classification rely on the high-level lexical and syntactic features obtained by NLP tools such as WordNet, dependency parser, part-of-speech (POS) tagger, and named entity recognizers (NER). In addition, state-of-the-art neural models based on attention mechanisms do not fully utilize information of entity that may be the most crucial features for relation classification. To address these issues, we propose a novel end-to-end recurrent neural model which incorporates an entity-aware attention mechanism with a latent entity typing (LET) method. Our model not only utilizes entities and their latent types as features effectively but also is more interpretable by visualizing attention mechanisms applied to our model and results of LET. Experimental results on the SemEval-2010 Task 8, one of the most popular relation classification task, demonstrate that our model outperforms existing state-of-the-art models without any high-level features.
BibTeX
@inproceedings{lee2019semantic,
  author = {Joohong Lee and Sangwoo Seo and Yong Suk Choi},
  title = {{Semantic Relation Classification via Bidirectional LSTM Networks with Entity-aware Attention using Latent Entity Typing}},
  booktitle = {Symmetry},
  year = {2019},
  url = {https://openreview.net/forum?id=Mrapz3un9H2},
}
Figure