Learning to Transfer Relational Representations through Analogy

Gaetano Rossiello, Alfio Gliozzo, Michael R. GlassAAAI 2019 · DOI

Abstract

This paper introduces "analogy embeddings"—a novel approach to representing semantic relationships between entities by training a model to recognize proportional analogies ($A:B = C:D$). By leveraging a hierarchical Siamese network (HSN) and distant supervision from large-scale knowledge bases, the authors create transferable relational representations that significantly boost the performance of state-of-the-art relation extraction models. This method effectively addresses the "long-tail" problem in relation extraction, where rare relations lack sufficient training data. Experimental results on standard benchmarks demonstrate that integrating analogy embeddings leads to substantial improvements in precision, recall, and F1 scores.

Introduction

Relation Extraction (RE) is the task of identifying semantic links (e.g., born_in, works_for) between entities in unstructured text. While deep learning has advanced this field, two major hurdles remain: the Long-Tail Problem, where rare relations lack the massive labeled data required by standard models, and Domain Specificity, where gathering human-annotated examples for new fields is prohibitively expensive.

This research shifts the paradigm from simple classification ("What relation is this?") to analogy detection ("Is the relationship between these two entities the same as the relationship between those two?"). By framing the problem as an analogy, the researchers can use distant supervision to generate a nearly infinite supply of training data without manual labeling.

Methods

The core idea is based on proportional analogies. If (Robert Plant, Led Zeppelin) and (David Gilmour, Pink Floyd) both share the relation member_of_band, then these pairs are analogous.

Overview of the Hierarchical Siamese Network for learning analogy embeddings.

To learn these analogies, the authors designed a Hierarchical Siamese Network (HSN):

  • Multi-Instance Learning: Since an entity pair might appear in many different sentences (mentions), the model processes a set of mentions for each pair to handle linguistic variety.
  • Hierarchical Attention: The model uses Gated Recurrent Units (GRUs) with attention at the word level (to find important words) and the mention level (to identify which sentence best describes the relationship).
  • Training Objective: The network is trained to minimize the distance in vector space between analogous pairs and maximize the distance between non-analogous pairs.

Once trained, the HSN acts as a feature extractor. These "analogy embeddings" are then integrated into existing state-of-the-art relation extraction models (such as PCNN-KI), providing them with a pre-trained understanding of how relations are typically expressed in language.

Experiments and Results

The model was evaluated on several key datasets, including NYT-FB (New York Times articles mapped to Freebase) and CC-DBP (Common Crawl and DBpedia). The integration of analogy embeddings led to clear improvements across the board.

Precision-Recall curves on the NYT-FB dataset showing the performance of analogy embeddings compared to baselines.

Key findings include:

  • Performance Gains: On the NYT-FB dataset, the PCNN-KI+Analogy model showed a significant lift in the Precision-Recall curve compared to previous state-of-the-art methods like PCNN+ATT.
  • Metric Improvements: On the CC-DBP dataset, the authors reported a jump in Area Under Curve (AUC) from 0.437 to 0.500 and an F1 score increase from 0.468 to 0.504 when adding analogy embeddings to the baseline model.
  • Relational Paraphrasing: The model effectively learns that different linguistic patterns (e.g., "X is the singer of Y" vs "Y's frontman X") represent the same underlying semantic relation.
  • Generalization: The analogy embeddings proved robust; even when the HSN was trained on one corpus (Wikipedia/T-REX), the features it produced were highly effective when applied to a completely different corpus (NYT-FB).

Conclusion

By representing relations through analogies, this work provides a powerful mechanism for knowledge transfer in NLP. While the hierarchical attention mechanism is computationally intensive, the resulting embeddings significantly improve the extraction of rare relations. Future work will explore these embeddings for other tasks, such as Question Answering and Knowledge Base Population.

Summary updated

BibTeX
@inproceedings{rossiello2019learning,
  author = {Gaetano Rossiello and Alfio Gliozzo and Michael R. Glass},
  title = {{Learning to Transfer Relational Representations through Analogy}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2019},
  doi = {10.1609/aaai.v33i01.330110015},
}
Figure