Distantly-Supervised Neural Relation Extraction with Side Information using BERT

Johny Moreira, Chaina Oliveira, David Macêdo, Cleber Zanchettin, Luciano Barbosa

TL;DR

The researchers introduced BERT-Side, a simplified yet more powerful model for Distantly-Supervised Relation Extraction. By replacing complex syntactic encoders with pre-trained BERT embeddings while retaining "side information" (like entity types and relation aliases), the model achieves near-perfect precision on standard benchmarks, outperforming previous state-of-the-art methods like RESIDE.

What Problem Does This Paper Address?

Relation Extraction (RE) is the task of identifying semantic relationships between entities in text (e.g., recognizing that "Barack Obama was born in Honolulu" implies the relation bornIn). To avoid the high cost of manual labeling, researchers use Distant Supervision (DS), which automatically aligns existing Knowledge Bases (KBs) with raw text.

However, DS is notoriously noisy. If a KB says "Obama" and "USA" have a bornIn relationship, DS assumes every sentence containing both entities expresses that relation—which isn't true (e.g., "Obama ran for the Senate in the USA").

Previous state-of-the-art models like RESIDE tried to solve this by using "side information" (extra data from KBs) and complex neural architectures involving Bi-GRUs and Graph Convolutional Networks (GCNs) to understand sentence structure. The authors of this paper argue that these complex syntactic modules can be simplified and improved by leveraging the contextual power of BERT.

Methods

The authors propose BERT-Side, which adapts the RESIDE framework but swaps out its heavy "Syntactic Sentence Encoding" module.

  1. Sentence Encoding with BERT: Instead of using Bi-GRUs and GCNs over dependency trees, the model uses pre-trained BERT embeddings. Specifically, it uses the [CLS] token representation, pooled from the second-to-last hidden layers, to capture the semantic essence of the sentence.
  2. Side Information Acquisition: The model retains RESIDE’s "side information" strategy. It incorporates:
  3. Entity Types: Using embeddings for entity categories (e.g., /person, /location).
  4. Relation Aliases: Using Open IE to find phrases that act as aliases for relations (e.g., "is the author of" as an alias for developerOf).
  5. Instance Set Aggregation: Since multiple sentences might mention the same entity pair (a "bag"), the model uses an attention mechanism to weigh which sentences most likely express the target relation.
  6. Classification: The final representation is passed through two fully connected layers with a softmax function to predict the relation.

Datasets

The model was evaluated on two major benchmarks:

  • Google Distant Supervision (GDS): An extension of the Google relation extraction corpus containing 5 relations.
  • Riedel (NYT) Dataset: A widely used dataset created by aligning Freebase relations with New York Times articles (containing 53 relations).

Key Insights and Ideas

  • Context Trumps Syntax: The most significant "aha moment" is that BERT’s contextual embeddings are so rich that they can replace explicit syntactic features (like dependency trees and GCNs) without losing performance. This significantly simplifies the model architecture.
  • The Precision Powerhouse: BERT-Side is exceptionally good at identifying correct relations when it is confident. It achieved a Precision@100 of 1.00 across almost all test scenarios, meaning its top 100 predictions were virtually always correct.
  • Side Info is Still Vital: Even with a powerful language model like BERT, adding external KB "side information" (entity types and aliases) helps the model navigate the noise inherent in distant supervision.

Key Results

  • Superior Precision: On the GDS and Riedel datasets, BERT-Side outperformed the BGWA and RESIDE baselines in P@N (Precision at N) metrics. For example, in the Riedel dataset, BERT-Side maintained 1.00 precision for P@100, P@200, and P@300, whereas RESIDE hovered between 0.70 and 0.80.
  • The AUC Trade-off: While precision was high, the Area Under the Curve (AUC) was lower than expected, particularly on the Riedel dataset. This is because the dataset is highly unbalanced, dominated by the "NA" (no relation) class. While BERT-Side is very accurate for positive relations, the sheer volume of "NA" instances impacts the overall recall curve.

Limitations and Future Work

  • Class Imbalance: The authors acknowledge that the model struggles with the "NA" relation dominance in large datasets like Riedel. The high precision comes at the cost of a steeper drop in the Precision-Recall curve as recall increases.
  • Feature Extraction vs. Fine-tuning: In this study, BERT was used as a fixed feature extractor (bert-as-service). The authors suggest that fine-tuning the BERT model specifically for the relation extraction task could yield even better results.
  • Objective Functions: Future work will explore different loss functions or objective functions specifically designed to handle the extreme class imbalance found in distantly supervised data.

Summary updated

Abstract

Relation extraction (RE) consists in categorizing the relationship between entities in a sentence. A recent paradigm to develop relation extractors is Distant Supervision (DS), which allows the automatic creation of new datasets by taking an alignment between a text corpus and a Knowledge Base (KB). KBs can sometimes also provide additional information to the RE task. One of the methods that adopt this strategy is the RESIDE model, which proposes a distantly-supervised neural relation extraction using side information from KBs. Considering that this method outperformed state-of-the-art baselines, in this paper, we propose a related approach to RESIDE also using additional side information, but simplifying the sentence encoding with BERT embeddings. Through experiments, we show the effectiveness of the proposed method in Google Distant Supervision and Riedel datasets concerning the BGWA and RESIDE baseline methods. Although Area Under the Curve is decreased because of unbalanced datasets, P@N results have shown that the use of BERT as sentence encoding allows superior performance to baseline methods.
BibTeX
@inproceedings{moreira2020distantly,
  author = {Johny Moreira and Chaina Oliveira and David Macêdo and Cleber Zanchettin and Luciano Barbosa},
  title = {{Distantly-Supervised Neural Relation Extraction with Side Information using BERT}},
  booktitle = {IJCNN},
  year = {2020},
  url = {https://openreview.net/forum?id=KP69R44Pr0E},
}

Loading related content...

Figure