Relation Adaptation: Learning to Extract Novel Relations with Minimum Supervision

Danushka Bollegala, Yutaka Matsuo, Mitsuru IshizukaIJCAI 2011 · OpenReview

Abstract

Extracting the relations that exist between two entities is an important step in numerous Web-related tasks such as information extraction. A supervised relation extraction system that is trained to extract a particular relation type might not accurately extract a new type of a relation for which it has not been trained. However, it is costly to create training data manually for every new relation type that one might want to extract. We propose a method to adapt an existing relation extraction system to extract new relation types with minimum supervision. Our proposed method comprises two stages: learning a lower-dimensional projection between different relations, and learning a relational classifier for the target relation type with instance sampling. We evaluate the proposed method using a dataset that contains 2000 instances for 20 different relation types. Our experimental results show that the proposed method achieves a statistically significant macro-average F-score of 62.77. Moreover, the proposed method outperforms numerous baselines and a previously proposed weakly-supervised relation extraction method.

Introduction

Information Extraction (IE) aims to identify semantic links between entities, such as recognizing that "Steve Jobs" and "Apple" are connected by a ceoOf relation. Traditional supervised learning approaches require thousands of manually labeled examples for each relation type, creating a significant bottleneck given the vast number of potential relations on the web. A model trained on financial news (e.g., "mergers and acquisitions") typically fails to generalize to other domains like cinema (e.g., "directors"). This paper addresses the Relation Adaptation problem: adapting a system trained on several "source" relations to a new "target" relation using only a minimal set of training instances.

Methodology

The proposed framework consists of two primary stages: Relational Mapping and One-sided Under-sampling.

Relational Mapping

The system extracts lexical and syntactic patterns from text and categorizes them into Relation-Specific (RS) patterns, which are unique to a single relation, and Relation-Independent (RI) patterns, which are shared across multiple relations. For example, "X directed Y" can apply to both corporate leadership and film production.

Conceptual mapping between relation-specific and relation-independent patterns.

The method identifies RI patterns using Shannon entropy; patterns appearing across diverse entity pairs are treated as general "pivot" patterns. By constructing a bipartite graph connecting RS and RI patterns and applying spectral clustering, the system projects high-dimensional features into a lower-dimensional latent space. This allows similar relations to be grouped together even when they utilize different surface-level phrasing.

One-sided Under-sampling

To combat class imbalance—where source data vastly outweighs the few target examples—the authors introduced a custom under-sampling algorithm. This intelligently selects source data subsets most relevant to the target, preventing the target relation signal from being overwhelmed during training.

Experimental Setup

The method was evaluated using 20 relation types from the YAGO Ontology, with 100 entity pairs per relation. Web contexts were retrieved via the Yahoo BOSS API, yielding approximately 7,000 snippets per pair. To simulate a low-supervision environment, models were trained with 60 instances for source relations but only 10 instances for the target relation.

Results and Discussion

The experimental results demonstrate that the proposed method significantly outperforms existing approaches.

Performance comparison of the proposed method against various baselines.

As shown in the performance table, the proposed "Comb.

  • Samp." method achieved a macro-average F-score of 62.77, notably higher than the next best baseline (57.62) and the state-of-the-art cross-domain method by Jiang (2009) at 55.62. The method was particularly effective in precision (86.47), which is critical for maintaining the integrity of automated knowledge bases.

Key insights include:

  • RI Patterns as Pivots: Shared linguistic "skeletons" allow for effective knowledge transfer between relations.
  • Feature Selection: Using all available patterns can degrade performance; filtering for high-entropy, independent patterns creates a more robust bridge for transfer.
  • Sampling Necessity: Without under-sampling, precision drops as the volume of source data drowns out the target examples.

Conclusion

This research presents a robust method for adapting relation extraction systems to novel types with minimal supervision. While entropy-based selection can occasionally introduce noise, the dual-stage approach of latent mapping and intelligent sampling provides a significant performance boost. Future work may explore scaling this method to thousands of relations and applying the adaptation logic to other classification tasks.

Summary updated

Abstract

Extracting the relations that exist between two entities is an important step in numerous Web-related tasks such as information extraction. A supervised relation extraction system that is trained to extract a particular relation type might not accurately extract a new type of a relation for which it has not been trained. However, it is costly to create training data manually for every new relation type that one might want to extract. We propose a method to adapt an existing relation extraction system to extract new relation types with minimum supervision. Our proposed method comprises two stages: learning a lower-dimensional projection between different relations, and learning a relational classifier for the target relation type with instance sampling. We evaluate the proposed method using a dataset that contains 2000 instances for 20 different relation types. Our experimental results show that the proposed method achieves a statistically significant macro-average F-score of 62.77. Moreover, the proposed method outperforms numerous baselines and a previously proposed weakly-supervised relation extraction method.
BibTeX
@inproceedings{bollegala2011relation,
  author = {Danushka Bollegala and Yutaka Matsuo and Mitsuru Ishizuka},
  title = {{Relation Adaptation: Learning to Extract Novel Relations with Minimum Supervision}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2011},
  url = {https://openreview.net/forum?id=Sk4FvVM_bS},
}
Figure