Corpus-Based Task-Specific Relation Discovery

Karthik Venkat RamananACL 2023

TL;DR

This paper introduces a new task called "corpus-based task-specific relation discovery," aiming to find novel relationships between entity types within a given corpus, beyond predefined schemas. They propose a novel end-to-end framework that leverages prompting and clustering to achieve this, significantly outperforming existing methods. This work is crucial for uncovering deeper insights from large text collections and improving downstream NLP tasks.

What Problem Does This Paper Address?

Traditional relation extraction (RE) systems are limited because they rely on a fixed set of predefined relations. This "closed-world" approach makes it impossible to discover new, emerging, or domain-specific relationships that aren't part of the initial schema. While Open Information Extraction (OpenIE) attempts to extract relations without a schema, its output is often too general, noisy, or redundant, requiring significant post-processing. The authors identify a gap in discovering task-specific relations from a corpus, meaning relations between specific types of entities (e.g., viral variants and geographic locations) that are relevant to a particular task or domain, without prior knowledge of what those relations might be.

Methods

The proposed methodology consists of three main steps:

  1. Identifying Candidate Spans: They use a combination of Named Entity Recognition (NER) and zero-shot entity linking (using BLINK) to identify entities within a paragraph. By linking these entities to Wikipedia and then Wikidata, they can determine their semantic types. If entities of the specified "head" and "tail" semantic types co-occur in a paragraph, it's marked as a candidate span for relation discovery.
  2. Identifying Novel Relation Phrases: This is the core of their novel approach. They employ a self-supervised prompting strategy using the BART encoder-decoder transformer architecture. The model is fine-tuned on existing data (like Wikidata) to generate relation phrases that explain the relationship between the identified head and tail entities. They experiment with various prompting strategies, with "Prompt-del" showing the most promising results.
  3. Clustering Relation Phrases: To group similar discovered relation phrases, they use a k-means clustering approach. Crucially, they incorporate the head and tail semantic types into the clustering process. This helps disambiguate phrases that might have the same surface form but represent different relations depending on the entity types involved (e.g., "written by" could mean "screenwriter" for a movie or "author" for a book). They generate "pseudosentences" using these semantic types and the relation phrase to create embeddings for clustering.

Datasets

  • Wiki-Art: A new dataset introduced by the authors, created by scraping documents from Wikipedia pages related to movies and comic books. It's designed for task-specific relation discovery.
  • REBEL dataset: Used for distant supervision during the training of their prompting model.

Key Insights and Ideas

  • Task-Specific Relation Discovery is a Novel and Important Task: The ability to discover relations relevant to a specific task (defined by entity types) is a significant step beyond general OpenIE or fixed-schema RE.
  • Prompting is Powerful for Relation Phrase Generation: Fine-tuning language models with specific prompts can effectively elicit novel relation phrases from text.
  • Semantic Type Information is Crucial for Clustering: Disambiguating relation phrases based on the semantic types of the entities involved is key to forming coherent clusters and accurate relation discovery.
  • Existing OpenIE Methods Struggle with Specificity: Baselines like Stanford OpenIE and OpenIE5 tend to produce overly general relations, making them less useful for detailed analysis.

Key Results

  • Relation Identification: Their "Prompt-del" model achieved an accuracy of 0.90 on relation identification, significantly outperforming Stanford OpenIE (0.11) and OpenIE5 (0.17).
  • Unsupervised Relation Discovery: On unsupervised relation discovery, their model achieved an Adjusted Rand Index (ARI) of 0.69 and a Permutation Accuracy (ACC) of 0.85, substantially higher than baselines.
  • Corpus-Based Relation Discovery: Their end-to-end pipeline successfully discovered 15 out of 19 ground-truth relations in their corpus, demonstrating its effectiveness in a more realistic setting.
  • Qualitative Improvements: Visualizations (t-SNE) show that their method produces more coherent and well-separated clusters of discovered relations compared to baselines. They also highlight specific examples where their model correctly identifies relations missed by OpenIE (e.g., "voiced by," "colorist").

Limitations and Future Work

  • Domain Shift: The current prompting model relies on a training set, which might not generalize well to domains with significant shifts in relation expression. Future work could explore more fully unsupervised prompting approaches.
  • Limited Number of Relations: The study focused on a limited set of 25 relations. Future work aims to utilize Wikidata more comprehensively to discover a larger and more diverse set of relations.
  • Extending to Multiple Domains: The authors plan to extend their approach to multiple domains to identify and address potential gaps in their methodology.

Summary updated

BibTeX
@inproceedings{ramanan2023corpus,
  author = {Karthik Venkat Ramanan},
  title = {{Corpus-Based Task-Specific Relation Discovery}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
}
Figure