Extracting Multi-valued Relations from Language Models

Sneha Singhania, Simon Razniewski, Gerhard WeikumACL 2023

TL;DR

This paper tackles the challenge of extracting multi-valued relations (e.g., "Paris borders France, Germany, and Spain") from large language models (LMs), which typically struggle with this due to their tendency to predict single, most probable answers. The authors propose novel prompting strategies and selection mechanisms to improve the extraction of these complex relational facts, achieving a respectable 49.5% F1 score on this difficult task.

What Problem Does This Paper Address?

Large language models (LMs) have shown promise in implicitly storing factual knowledge. However, existing methods for extracting this knowledge, often using "cloze-style" prompts (like "Dante was born in [MASK]"), primarily focus on relations with a single correct answer. This is a significant limitation because many real-world relations are multi-valued. For instance, a country can border multiple other countries, or a musician can play several instruments. Extracting these multi-valued facts directly from LMs, without relying on external knowledge bases, is a crucial but underexplored area. The paper aims to bridge this gap by developing techniques to probe LMs for these more complex relational structures.

Methods

The core of the paper's methodology involves a two-phase approach:

  1. Candidate List Generation: The researchers probe LMs using various prompting techniques to generate a ranked list of potential objects that could fill a given relation slot. They evaluate existing prompt types (discrete and continuous) and introduce their own set of 50 "domain-knowledge-rich" manual prompts, designed to be more effective for multi-valued relations by incorporating specific linguistic variations and domain expertise.
  2. Object Selection: Since LM probabilities alone aren't always reliable indicators of factual accuracy, the paper explores several mechanisms to select the correct objects from the generated candidate list. These include:
  3. Top-k: Selecting the k most probable objects.
  4. Prob-x: Selecting objects with a probability above a certain threshold.
  5. Cumul-x: Selecting objects based on a cumulative probability threshold.
  6. Count Probe: Using the LM to predict the number of correct objects and then selecting that many top candidates.
  7. Verification Probe: Using the LM to factually verify each candidate triple (subject, relation, object) by asking a yes/no question.

Datasets

The experiments were conducted using the LAMA benchmark, specifically focusing on seven diverse multi-valued relations. For each relation, approximately 200 subjects were sampled, along with a complete list of objects from the Wikidata KB. The subjects were chosen based on popularity, as measured by their Wikidata ID and Twitter follower count for person-type subjects. The paper does not introduce a new benchmark dataset but rather adapts existing resources for the multi-valued relation extraction task.

Key Insights and Ideas

  • Multi-valued relations are hard for LMs: The significant gap between precision@1 and the maximum possible F1 score (max-F1) across various relations highlights the inherent difficulty LMs face in ranking all correct multi-valued objects above incorrect ones.
  • Prompt engineering matters, especially for multi-valued facts: The authors' custom-designed, domain-knowledge-rich prompts significantly outperform existing automated prompting methods in generating higher-quality candidate lists for multi-valued relations. This suggests that simply using generic prompts is insufficient.
  • Probability alone isn't enough: The paper demonstrates that LM probabilities are not perfectly calibrated for factual accuracy. This necessitates sophisticated selection mechanisms beyond simply picking the top-ranked items.
  • Selection mechanisms offer a trade-off: Different selection strategies (like top-k vs. prob-x vs. count probe) yield varying balances of precision and recall, indicating that the optimal choice depends on the specific relation and desired outcome.
  • LM size doesn't guarantee success: Larger LMs like T5 and BART, when probed with top-k selection, did not necessarily outperform BERT-large, suggesting that architectural differences and prompting strategies play a crucial role.

Key Results

  • The authors' proposed manual prompts achieved the best candidate lists in terms of macro-averaged max-F1 scores.
  • The best selection mechanism, a learned relation-specific threshold (similar to Prob-x), achieved a 49.5% F1 score on the multi-valued slot-filling task.
  • For the most challenging relations (parts of chemical compounds, official languages, instruments of musicians), their prompts outperformed the best baseline (SoftPrompts) by approximately 5% points in max-F1.
  • The best overall precision, recall, and F1 scores achieved by a selection mechanism were 54.1% precision, 50.8% recall, and 49.5% F1 score.

Limitations and Future Work

The paper acknowledges several limitations:

  • Incomplete Knowledge Extraction: Even with their best methods, LMs do not fully capture all correct multi-valued objects. The study found that a significant percentage of valid objects were either ranked too low or not generated at all.
  • Prompt Design Difficulty: Designing effective prompts for multi-valued relations remains a significant challenge, and the performance can vary considerably based on prompt design.
  • Calibration Issues: The probabilities generated by LMs are not perfectly calibrated to reflect factual accuracy, making direct reliance on them problematic.

Future work suggested by the authors includes:

  • Improving overall precision and recall for multi-valued relation extraction.
  • Investigating the impact of using LMs to fill gaps in existing knowledge bases.
  • Exploring more advanced prompting techniques and selection strategies.
  • Further research into calibrating LM outputs for factual knowledge.

Summary updated

BibTeX
@inproceedings{singhania2023extracting,
  author = {Sneha Singhania and Simon Razniewski and Gerhard Weikum},
  title = {{Extracting Multi-valued Relations from Language Models}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
}
Figure