Distilling Hypernymy Relations from Language Models: On the Effectiveness of Zero-Shot Taxonomy Induction

Devansh Jain, Luis Espinosa AnkeNAACL 2022

TL;DR

This paper explores how well large language models (LLMs) can perform zero-shot taxonomy induction, a task crucial for organizing knowledge. They demonstrate that simple prompt-based methods using LLMs can be surprisingly effective, often outperforming traditional methods and even achieving state-of-the-art results on certain benchmarks.

What Problem Does This Paper Address?

Taxonomy learning (TL) is the fundamental task of organizing concepts into hierarchical "is-a" relationships, like "a dog is a type of mammal." This is vital for many NLP tasks requiring reasoning and inference. Traditional TL methods often rely on hand-crafted features, complex algorithms, or external knowledge bases like WordNet, making them expensive and less adaptable. More recently, LLMs have shown promise in various NLP tasks without task-specific training (zero-shot learning). However, their effectiveness for zero-shot taxonomy induction, particularly by distilling knowledge through prompting, remained an open question. This paper aims to systematically evaluate this approach.

Methods

The authors investigate two primary methods for zero-shot taxonomy induction using LLMs:

  • MLM Prompting (Masked Language Model): This method leverages the "fill-in-the-blanks" capability of masked language models (like BERT). They construct prompts where a term is placed in a context, and the LLM predicts the masked word, inferring a hypernym. For example, "physics is a kind of [MASK]." They use a "RestrictMLM" approach, limiting the prediction to a fixed vocabulary, and also an unrestricted version.
  • LMScorer (Sentence Scoring): This method uses LLMs to score the naturalness or fluency of sentences that express a potential hypernymy relationship. Causal LMs (like GPT-2) and masked LMs are used to assign a score to sentences like "Trout is a type of fish." The term that results in the most fluent sentence is considered the hypernym.

They also explore different prompt templates (e.g., "is more general than," "is a type of") and analyze their impact.

Datasets

The paper evaluates their methods on existing taxonomy induction benchmarks:

  • TExEval-1: Specifically, the "equipment" taxonomy.
  • TExEval-2: The "environment," "science," and "food" taxonomies.

These datasets are chosen because they do not provide training data, making them suitable for unsupervised and zero-shot approaches.

Key Insights and Ideas

  • LLMs are surprisingly good at zero-shot TL: Even simple prompt-based methods can achieve competitive results, often surpassing traditional supervised or resource-intensive methods.
  • Prompt engineering matters: The choice of prompt template significantly impacts performance. Unambiguous prompts like "is a type of" are more effective than vaguer ones.
  • Frequency is a factor: Prompts that are more frequent in the LLM's pretraining data tend to perform better.
  • Single-token limitation: A key bottleneck for prompt-based methods is their inability to predict multi-token hypernyms directly, which can limit performance on datasets with many such terms.

Key Results

  • Competitive performance: In the "science" domain, their best model (RestrictMLM) outperformed CTP and was competitive with state-of-the-art methods like TaxoRL.
  • Outperforming baselines: Across larger datasets like "food" and "equipment," all three proposed approaches (PromptMLM, RestrictMLM, LMScorer) outperformed the best existing systems.
  • LM performance: Among MLMs, BERT-Large generally performed best. GPT-2 Medium was competitive with MLMs for sentence scoring.
  • Prompt impact: The "type" prompt consistently yielded better results than "gen" or "spec" prompts.

Limitations and Future Work

  • Single-token hypernyms: The current prompt-based methods are limited to predicting single-token hypernyms. This is a significant limitation, as many real-world taxonomies contain multi-token terms. The paper shows that filtering out multi-token hypernyms can actually increase F-score for these methods, highlighting the issue.
  • Future directions: The authors suggest exploring multilingual signals and integrating traditional word embeddings with contextual representations from LLMs. They also mention the potential for automatic prompt generation, though they did not pursue it due to a lack of training data.

Summary updated

BibTeX
@inproceedings{jain2022distilling,
  author = {Devansh Jain and Luis Espinosa Anke},
  title = {{Distilling Hypernymy Relations from Language Models: On the Effectiveness of Zero-Shot Taxonomy Induction}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2022},
}
Figure