Matching the Blanks: Distributional Similarity for Relation Learning

Livio Baldini Soares, Nicholas FitzGerald, Jeffrey Ling, Tom Kwiatkowski

TL;DR

The paper introduces "Matching the Blanks" (MTB), a self-supervised pre-training method that learns task-agnostic relation representations from text using only entity-linking information. By training a BERT-based model to determine whether two text snippets describe the same underlying relation—while masking out the entity names themselves—the approach achieves state-of-the-art performance on supervised benchmarks and even outperforms humans on few-shot relation matching without any task-specific training data.

What Problem Does This Paper Address?

Extracting relationships between entities from text (e.g., determining that "X was founded by Y" implies a founded_by relation) is a cornerstone of Information Extraction (IE). Traditional approaches fall into three flawed buckets:

  1. Supervised methods: Highly accurate but restricted to a small, rigid, predefined set of relations.
  2. Open IE: Highly scalable because it uses raw text strings as relations, but it lacks generality (e.g., it struggles to recognize that "is the author of" and "wrote" mean the same thing).
  3. Universal Schemas: Successfully bridge text and structured schemas but rely heavily on massive, pre-existing knowledge graphs (like Freebase) aligned to text.

The authors address the challenge of learning general-purpose, continuous representations of relations directly from raw text without relying on human annotations or structured knowledge graphs.

Methods

The authors build on top of BERT (BERT_LARGE) and introduce two key innovations:

1. Optimal Architecture for Relation Extraction

The authors systematically evaluate how to feed entity information into BERT and extract a fixed-length relation vector. The winning configuration uses Entity Markers at the input layer and Entity Start State at the output layer:

  • Input: Special boundary tokens ([E1], [/E1], [E2], [/E2]) are wrapped directly around the entity mentions in the text stream.
  • Output: The final relation representation is a concatenation of the hidden states corresponding to the [E1] and [E2] start tokens.
Variants of BERT architectures for relation extraction

2. Matching the Blanks (MTB) Pre-training

Based on an extension of Harris's distributional hypothesis—which posits that pairs of text segments containing the same entities are likely to express the same semantic relation—the authors design a binary classification task.

If two text segments share the same pair of entities, they form a positive pair; otherwise, they form a negative pair. Crucially, to prevent the model from simply memorizing entity names, they randomly replace entity text with a [BLANK] token with a probability of $\alpha = 0.7$. This forces the model to focus entirely on the contextual text surrounding the entities.

Datasets

  • Pre-training Corpus: 600 million relation statement pairs extracted from English Wikipedia paragraphs, automatically annotated using the Google Cloud Natural Language API for entity linking.
  • Evaluation Benchmarks:
  • FewRel: A few-shot relation classification dataset (evaluated on 5-way and 10-way configurations).
  • SemEval 2010 Task 8: A standard supervised relation classification dataset containing 19 relation types.
  • KBP37: A supervised dataset with 37 relation types.
  • TACRED: A large-scale supervised relation extraction dataset with 42 relation types.

Key Insights and Ideas

  • The Power of Blanks: Simply matching entities would cause the model to overfit to entity identities rather than learning abstract relations. Replacing entities with [BLANK] forces the network to learn the semantic meaning of the context.
  • Structural Markers Beat Positional Embeddings: Forcing BERT to see explicit boundary tokens ([E1], [E2]) in the text stream works significantly better than adding segment or positional embeddings to identify entity spans.
  • Zero-Shot Superiority: The self-supervised MTB representations are so robust that they outperform existing fully supervised models on few-shot tasks without ever seeing a single training example from those tasks.

Key Results

  • Few-Shot Matching (FewRel): Without any task-specific fine-tuning, MTB achieved 93.9% accuracy on the 5-way 1-shot task and 89.2% on the 10-way 1-shot task. This significantly outperformed the previous supervised state-of-the-art (ProtoNet at 69.2% and 56.4%) and even surpassed human performance (92.2% and 85.9%).
  • Supervised Fine-Tuning: When fine-tuned on standard datasets, MTB set new state-of-the-art F1 scores across the board:
  • SemEval 2010 Task 8: 89.5% (vs. previous SOTA of 84.8%)
  • KBP37: 69.3% (vs. previous SOTA of 58.8%)
  • TACRED: 71.5% (vs. previous SOTA of 68.2%)
  • Data Efficiency: In low-resource settings (using only 1% or 10% of available training data), the performance gap between MTB and standard BERT widened dramatically, proving MTB's high utility when labeled data is scarce.

Limitations and Future Work

  • Entity Linker Dependency: The pre-training pipeline relies on an off-the-shelf entity linker to construct the corpus. Consequently, errors or biases in entity resolution could propagate into the learned relation representations.
  • Future Directions: The authors propose using MTB representations for unsupervised relation discovery via clustering (grouping similar relation statements together without predefined labels). They also suggest exploring how these continuous relation vectors can be integrated directly into distributed knowledge bases.

Summary updated

Abstract

General purpose relation extractors, which can model arbitrary relations, are a core aspiration in information extraction. Efforts have been made to build general purpose extractors that represent relations with their surface forms, or which jointly embed surface forms with relations from an existing knowledge graph. However, both of these approaches are limited in their ability to generalize. In this paper, we build on extensions of Harris' distributional hypothesis to relations, as well as recent advances in learning text representations (specifically, BERT), to build task agnostic relation representations solely from entity-linked text. We show that these representations significantly outperform previous work on exemplar based relation extraction (FewRel) even without using any of that task's training data. We also show that models initialized with our task agnostic representations, and then tuned on supervised relation extraction datasets, significantly outperform the previous methods on SemEval 2010 Task 8, KBP37, and TACRED.
BibTeX
@inproceedings{soares2019matching,
  author = {Livio Baldini Soares and Nicholas FitzGerald and Jeffrey Ling and Tom Kwiatkowski},
  title = {{Matching the Blanks: Distributional Similarity for Relation Learning}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
  url = {https://openreview.net/forum?id=4za_85yCZE},
}
Figure