Semantic Relation Classification via Bidirectional LSTM Networks with Entity-Aware Attention Using Latent Entity Typing

Joohong Lee, Sangwoo Seo, Yong Suk Choi

TL;DR

Researchers have developed a novel end-to-end neural network that classifies the relationship between two entities in a sentence (e.g., "The crash was caused by the attack" $\rightarrow$ Cause-Effect) without relying on external, error-prone NLP tools like dependency parsers. By combining a Bidirectional LSTM with a specialized "entity-aware" attention mechanism and Latent Entity Typing (LET), the model achieves a state-of-the-art 85.2% F1-score on the SemEval-2010 benchmark.

What Problem Does This Paper Address?

Relation Classification (RC) is a core task in Information Extraction. The goal is to identify the semantic relationship between two tagged entities in a sentence. Traditionally, this relied on "high-level" features from external tools (Part-of-Speech taggers, Named Entity Recognizers, and Dependency Parsers).

However, this approach has two major flaws:

  1. Error Propagation: If the dependency parser makes a mistake, the relation classifier is almost guaranteed to fail.
  2. Computational Cost: Running a full suite of NLP tools for every sentence is slow and expensive.

While recent "end-to-end" neural models have tried to solve this using raw text, they often treat all words equally or use generic attention mechanisms that don't fully leverage the most important information in the sentence: the entities themselves and their specific categories.

Methods

The authors propose a four-stage architecture:

  1. Word Representation: Words are converted into vectors using pre-trained GloVe embeddings.
  2. Self-Attention: A multi-head attention layer (similar to the Transformer architecture) captures the correlation between words, helping the model understand context before processing the sequence.
  3. Bidirectional LSTM (BiLSTM): This layer captures the sequential nature of the sentence from both directions, producing a hidden state for every word.
  4. Entity-aware Attention with Latent Entity Typing (LET): This is the paper's primary innovation. Instead of just looking at the whole sentence, the attention mechanism specifically incorporates:
  5. Relative Position Features: How far each word is from Entity 1 and Entity 2.
  6. Latent Entity Typing (LET): Since the model doesn't use an external NER tool, it learns to categorize entities into "latent types" (clusters) automatically. It represents each entity as a weighted combination of $K$ learned type vectors.

The final "sentence representation" is a weighted sum of the BiLSTM hidden states, where the weights are determined by how relevant a word is to the specific entities and their latent types.

Datasets

The model was evaluated on the SemEval-2010 Task 8 dataset.

  • Content: 10,717 annotated sentences (8,000 for training, 2,717 for testing).
  • Task: Classify pairs into 9 directed relations (like Cause-Effect or Component-Whole) and one undirected Other class, totaling 19 possible labels.

Key Insights and Ideas

  • Entities are Anchors: The authors argue that the entities are the most powerful hints for the task. Even without reading the rest of the sentence, knowing that the entities are "crash" and "attack" strongly suggests a Cause-Effect relationship.
  • Latent Types > Hard Tags: Instead of using rigid categories like "Person" or "Organization" from an NER tool, LET allows the model to learn its own semantic clusters. In their visualization, the model grouped entities into meaningful categories: Type 1 (jobs/food), Type 2 (machinery/engineering), and Type 3 (disasters/negative concepts).
  • Interpretability through Visualization: By visualizing the attention weights, the authors show the model actually "looks" at the right words. For a Component-Whole relation, the model puts the highest weight on the word "inside."

Key Results

  • Performance: The model achieved an 85.2% F1-score.
  • Comparison: This outperformed standard BiLSTMs (82.7%) and even complex models that use WordNet and dependency features (84.3%).
  • Ablation: Adding Latent Entity Typing provided a 0.5% boost over the model using only entity-aware attention, proving that "soft" typing helps the model generalize.
  • Efficiency: Because it is end-to-end, it avoids the overhead of pre-processing sentences with heavy linguistic toolkits.

Limitations and Future Work

  • Fixed Latent Types: The authors chose $K=3$ for the number of latent types based on empirical testing. A more dynamic way to determine the number of types could be beneficial.
  • Dataset Scope: The model was only tested on the SemEval-2010 dataset. Its performance on more specialized domains (like medical or legal texts) remains to be seen.
  • Future Directions: The authors suggest extending this "entity-aware" approach to other tasks like Question Answering or Knowledge Base Population, where understanding entity roles is critical.

Summary updated

Abstract

Classifying semantic relations between entity pairs in sentences is an important task in natural language processing (NLP). Most previous models applied to relation classification rely on high-level lexical and syntactic features obtained by NLP tools such as WordNet, the dependency parser, part-of-speech (POS) tagger, and named entity recognizers (NER). In addition, state-of-the-art neural models based on attention mechanisms do not fully utilize information related to the entity, which may be the most crucial feature for relation classification. To address these issues, we propose a novel end-to-end recurrent neural model that incorporates an entity-aware attention mechanism with a latent entity typing (LET) method. Our model not only effectively utilizes entities and their latent types as features, but also builds word representations by applying self-attention based on symmetrical similarity of a sentence itself. Moreover, the model is interpretable by visualizing applied attention mechanisms. Experimental results obtained with the SemEval-2010 Task 8 dataset, which is one of the most popular relation classification tasks, demonstrate that our model outperforms existing state-of-the-art models without any high-level features.
BibTeX
@inproceedings{lee2019semantic,
  author = {Joohong Lee and Sangwoo Seo and Yong Suk Choi},
  title = {{Semantic Relation Classification via Bidirectional LSTM Networks with Entity-Aware Attention Using Latent Entity Typing}},
  booktitle = {Symmetry},
  year = {2019},
  url = {https://openreview.net/forum?id=Mrapz3un9H2},
}

Loading related content...

Figure