Distantly-Supervised Neural Relation Extraction with Side Information using BERT

Johny Moreira, Chaina Oliveira, David Macedo, Cleber Zanchettin, Luciano Barbosa

Abstract

Relation extraction (RE) consists in categorizing the relationship between entities in a sentence. A recent paradigm to develop relation extractors is Distant Supervision (DS), which allows the automatic creation of new datasets by taking an alignment between a text corpus and a Knowledge Base (KB). KBs can sometimes also provide additional information to the RE task. One of the methods that adopt this strategy is the RESIDE model, which proposes a distantly-supervised neural relation extraction using side information from KBs. Considering that this method outperformed state-of-the-art baselines, in this paper, we propose a related approach to RESIDE also using additional side information, but simplifying the sentence encoding with BERT embeddings. Through experiments, we show the effectiveness of the proposed method in Google Distant Supervision and Riedel datasets concerning the BGWA and RESIDE baseline methods. Although Area Under the Curve is decreased because of unbalanced datasets, P@N results have shown that the use of BERT as sentence encoding allows superior performance to baseline methods.

Introduction

Relation Extraction (RE) is the task of identifying semantic relationships between entities in text, such as recognizing that "Barack Obama was born in Honolulu" implies the relation bornIn. To avoid the high cost of manual labeling, researchers often use Distant Supervision (DS), which automatically aligns existing Knowledge Bases (KBs) with raw text.

However, DS is notoriously noisy. If a KB indicates that "Obama" and "USA" have a bornIn relationship, DS assumes every sentence containing both entities expresses that relation, which is often incorrect (e.g., "Obama ran for the Senate in the USA"). Previous state-of-the-art models like RESIDE addressed this by incorporating "side information" from KBs and using complex neural architectures like Bi-GRUs and Graph Convolutional Networks (GCNs) to model sentence structure. This paper proposes BERT-Side, which simplifies these complex syntactic modules by leveraging the contextual power of BERT.

Illustration of Distant Supervision: Entity pairs are mapped to relations in a KB, resulting in 'bags' of sentences ($S_i$) where only some (in red) actually express the relation.

Methods

BERT-Side adapts the RESIDE framework but replaces its heavy syntactic sentence encoding module with pre-trained BERT embeddings. The model architecture consists of several key components:

  1. Sentence Encoding with BERT: Instead of Bi-GRUs and GCNs over dependency trees, the model uses the [CLS] token representation from BERT, pooled from the second-to-last hidden layers, to capture sentence semantics.
  2. Side Information Acquisition: The model retains RESIDE’s strategy of incorporating external data:
  3. Entity Types: Embeddings for categories like /person or /location.
  4. Relation Aliases: Phrases identified via Open IE that act as aliases for relations (e.g., "is the author of").
  5. Instance Set Aggregation: Since multiple sentences might mention the same entity pair (forming a "bag"), an attention mechanism weighs which sentences most likely express the target relation.
  6. Classification: The final representation passes through two fully connected layers with a softmax function to predict the relation.

Results and Discussion

The model was evaluated on the Google Distant Supervision (GDS) and Riedel (NYT) datasets. BERT-Side demonstrated superior precision (P@N) compared to BGWA and RESIDE baselines. Notably, on the Riedel dataset, BERT-Side maintained a precision of 1.00 for P@100, P@200, and P@300, significantly outperforming RESIDE's 0.70–0.80 range.

However, the Area Under the Curve (AUC) was lower than expected on the Riedel dataset. This is attributed to extreme class imbalance, where the "NA" (no relation) class dominates. While BERT-Side is highly accurate for positive relations, the volume of "NA" instances impacts the overall recall curve. These results suggest that BERT’s contextual embeddings are powerful enough to replace explicit syntactic features, though handling dataset imbalance remains a challenge for future work, potentially through fine-tuning or specialized loss functions.

Summary updated

Abstract

Relation extraction (RE) consists in categorizing the relationship between entities in a sentence. A recent paradigm to develop relation extractors is Distant Supervision (DS), which allows the automatic creation of new datasets by taking an alignment between a text corpus and a Knowledge Base (KB). KBs can sometimes also provide additional information to the RE task. One of the methods that adopt this strategy is the RESIDE model, which proposes a distantly-supervised neural relation extraction using side information from KBs. Considering that this method outperformed state-of-the-art baselines, in this paper, we propose a related approach to RESIDE also using additional side information, but simplifying the sentence encoding with BERT embeddings. Through experiments, we show the effectiveness of the proposed method in Google Distant Supervision and Riedel datasets concerning the BGWA and RESIDE baseline methods. Although Area Under the Curve is decreased because of unbalanced datasets, P@N results have shown that the use of BERT as sentence encoding allows superior performance to baseline methods.
BibTeX
@inproceedings{moreira2020distantly,
  author = {Johny Moreira and Chaina Oliveira and David Macedo and Cleber Zanchettin and Luciano Barbosa},
  title = {{Distantly-Supervised Neural Relation Extraction with Side Information using BERT}},
  booktitle = {IJCNN},
  year = {2020},
  url = {https://openreview.net/forum?id=KP69R44Pr0E},
}
Figure