Clustering-based Unsupervised Generative Relation Extraction

Chenhan Yuan, Ryan A. Rossi, Andrew Katz, Hoda EldardiryarXiv 2020 · OpenReview

TL;DR

This paper introduces CURE, a novel unsupervised relation extraction framework that leverages self-supervised learning within an encoder-decoder architecture to better capture correlations between sentences describing the same entity pair. By training on predicting dependency graph shortest paths, CURE achieves state-of-the-art performance on standard datasets, outperforming existing unsupervised methods.

What Problem Does This Paper Address?

Constructing knowledge graphs (KGs) is crucial for many AI applications, and a key step is extracting relational triplets (subject, relation, object) from text. While supervised methods exist, they require extensive labeled data, which is often unavailable. Unsupervised relation extraction (RE) aims to overcome this by clustering similar relations without explicit labels. However, existing unsupervised methods often fail to consider the valuable information present in multiple sentences that describe the same entity pair. This lack of correlation modeling can lead to suboptimal performance. Furthermore, some methods rely on predefined features or trigger words, which can introduce biases.

Methods

The proposed CURE framework tackles these challenges in two main stages:

  1. Relation Extractor Training (Self-Supervised Encoder-Decoder):
    • CURE utilizes an encoder-decoder architecture for self-supervised learning.
  2. The input to the model is a set of sentences containing the same entity pair.
  3. The core idea is to predict the "semantic shortest path" (SSP) between entities within the dependency parse tree of a sentence.
  4. The encoder learns to extract relation information from these SSPs across multiple sentences.
  5. The decoder then attempts to reconstruct one of the SSPs using the encoded information. This "predicting the path" task allows the model to learn meaningful relation representations without explicit labels.
  6. Crucially, by processing multiple sentences for the same entity pair, the model implicitly learns to leverage correlations between them, improving the robustness of the extracted relation information.
  7. Triplets Clustering and Labeling:
    • Once the relation extractor is trained, it's used to generate vector representations for entity pairs based on their SSPs.
  8. These vectors are then clustered using Hierarchical Agglomerative Clustering (HAC).
  9. A novel method for automatically labeling these clusters is introduced. It leverages word vector similarity (using pre-trained Word2Vec) and word frequency within the SSPs of each cluster to derive descriptive relation labels, aiming for more accurate and meaningful labels than simply picking the most common words.

Datasets

The paper evaluates CURE on two standard datasets:

  • New York Times (NYT) dataset: A widely used dataset for relation extraction. The authors use a preprocessed version, selecting 500K training and 5K testing sentences. Entity pairs in the test set are matched to Freebase.
  • United Nations Parallel Corpus (UNPC) dataset: A multilingual corpus chosen for its larger scale and different genre (political meeting records) to ensure model robustness. They use 3.2M training and 2.6K testing sentences, with entity pairs matched to YAGO.

Key Insights and Ideas

  • Leveraging Sentence Correlations is Key: The paper strongly argues that considering multiple sentences describing the same entity pair during training significantly improves relation extraction performance. This is a departure from many prior unsupervised methods.
  • Self-Supervised Learning via SSP Prediction: Using the task of predicting a dependency graph's shortest path as a self-supervised objective is an elegant way to train a relation extractor without labeled data. It forces the model to learn structural and semantic information relevant to relations.
  • Beyond "Most Common Words" for Labels: The proposed word vector similarity-based labeling approach offers a more nuanced and potentially accurate way to describe the discovered relations compared to simply selecting the most frequent words. This can lead to more interpretable results.

Key Results

  • State-of-the-Art Performance: CURE consistently outperforms three strong baseline methods (Rel-LDA, VAE, and Open-RE) on both the NYT and UNPC datasets.
  • Significant F-1 Gain: On the NYT dataset, CURE achieves an average F-1 gain of 10.47% over the baselines. On the UNPC dataset, the gain is 6.59%.
  • Improved Clustering: The Rand Index scores show that CURE's clustering performance is superior to the baselines on both datasets, indicating better grouping of entity pairs with similar relations.
  • Better Relation Labeling: Qualitative comparisons in Table 6 demonstrate that CURE's word vector similarity-based labeling method generates more descriptive and accurate labels for relations compared to the common words approach.

Limitations and Future Work

  • Potential for Over-Clustering: The authors note that on the UNPC dataset, CURE sometimes performs more detailed clustering than necessary, dividing general relations into finer-grained subsets (e.g., "president" or "ambassador" for "isPoliticianOf").
  • Syntactic Encoding: The paper suggests future work could explore more advanced methods for encoding syntactic structure, such as Graph Neural Networks (e.g., Tree-LSTM), to further improve model accuracy.
  • Variational Autoencoders for Correlations: They also propose exploring VAEs that specifically leverage correlations between sentences with similar entity pairs.

Summary updated

Abstract

This paper focuses on the problem of unsupervised relation extraction. Existing probabilistic generative model-based relation extraction methods work by extracting sentence features and using these features as inputs to train a generative model. This model is then used to cluster similar relations. However, these methods do not consider correlations between sentences with the same entity pair during training, which can negatively impact model performance. To address this issue, we propose a Clustering-based Unsupervised generative Relation Extraction (CURE) framework that leverages an "Encoder-Decoder" architecture to perform self-supervised learning so the encoder can extract relation information. Given multiple sentences with the same entity pair as inputs, self-supervised learning is deployed by predicting the shortest path between entity pairs on the dependency graph of one of the sentences. After that, we extract the relation information using the well-trained encoder. Then, entity pairs that share the same relation are clustered based on their corresponding relation information. Each cluster is labeled with a few words based on the words in the shortest paths corresponding to the entity pairs in each cluster. These cluster labels also describe the meaning of these relation clusters. We compare the triplets extracted by our proposed framework (CURE) and baseline methods with a ground-truth Knowledge Base. Experimental results show that our model performs better than state-of-the-art models on both New York Times (NYT) and United Nations Parallel Corpus (UNPC) standard datasets.
BibTeX
@misc{yuan2020clustering,
  author = {Chenhan Yuan and Ryan A. Rossi and Andrew Katz and Hoda Eldardiry},
  title = {{Clustering-based Unsupervised Generative Relation Extraction}},
  year = {2020},
  eprint = {2009.12681},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2009.12681},
}
Figure