Zero- and Few-Shots Knowledge Graph Triplet Extraction with Large Language Models

Andrea Papaluca, Daniel Krefl, Sergio Rodríguez Méndez, Artem Lensky, Hanna SuominenACL 2024

TL;DR

This paper introduces a novel pipeline that augments Large Language Models (LLMs) with external Knowledge Base (KB) information to improve their performance on Triplet Extraction (TE) tasks, especially in zero- and few-shot settings. By dynamically retrieving contextual triplets or sentence-triplet pairs from a KB, they demonstrate that LLMs can become competitive with traditional, fully supervised models, even with smaller model sizes.

What Problem Does This Paper Address?

Triplet Extraction (TE) is a crucial task in Natural Language Processing (NLP) for understanding the core meaning of sentences by identifying (subject, predicate, object) triplets. Current state-of-the-art (SOTA) methods rely heavily on supervised learning, requiring extensive labeled datasets. This dependence limits their generality and applicability in real-world scenarios where labeled data is scarce or specific to niche domains. While Large Language Models (LLMs) offer a promising alternative for few-shot and zero-shot learning, their performance in TE tasks has often been underwhelming compared to traditional models. This paper aims to bridge this gap by enhancing LLM performance in TE through the strategic integration of external knowledge.

Methods

The core contribution of this work is a pipeline that enriches LLM prompts with contextual information retrieved from a Knowledge Base (KB). For a given input sentence, the pipeline dynamically queries a KB to gather relevant information. This information is then incorporated into the LLM prompt in two primary ways:

  1. Context Triplets: A set of relevant triplets from the KB is directly provided to the LLM.
  2. Sentence-Triplets Pairs: More informative examples, consisting of sentences paired with their corresponding extracted triplets from the KB, are provided.

The KB retrieval process involves encoding triplets and sentences using a sentence encoder (MiniLM) and then finding the most similar items to the input sentence via cosine similarity. The LLM is then prompted with the original sentence augmented by this retrieved context. The authors experimented with different LLMs (GPT-2, Falcon, LLaMA) and evaluated their performance in zero-shot, 2-shot, 0.5-shot (with KB triplets), and 5-shot (with KB sentence-triplets pairs) settings.

Datasets

The paper evaluates its approach on two standard TE datasets:

  • WebNLG: Originally designed for Natural Language Generation (NLG), it has been adapted for TE. The authors used the version revised by Zheng et al. (2017).
  • New York Times (NYT): This dataset is created through distant supervision, aligning NYT articles with the Freebase KB.

The training and validation splits of these datasets were used to construct the KB for retrieval.

Key Insights and Ideas

  • KB Augmentation is Key: Providing LLMs with external KB context significantly boosts their Triplet Extraction performance, making them competitive even with smaller models.
  • Sentence-Triplets Pairs are More Informative: Augmenting prompts with sentence-triplet pairs from the KB yields better results than just providing context triplets, suggesting LLMs benefit more from concrete examples of how relations are expressed in sentences.
  • LLM Scaling is Logarithmic, KB Quality is Linear: The performance of LLMs in TE scales logarithmically with their size (number of parameters). However, the performance scales linearly with the quality and completeness of the retrieved KB context. This suggests that investing in better KB retrieval mechanisms might be more impactful than simply using larger LLMs.
  • Dataset Generality Concerns: The high overlap between training, validation, and test sets in the datasets used suggests that current benchmarks might not be ideal for evaluating the true generalization capabilities of TE models.

Key Results

  • Improved Performance: The proposed KB-aided pipeline substantially improves LLM performance in TE, especially in few-shot settings.
  • Competitiveness: With KB augmentation, LLMs become competitive with classical NLP baselines, and larger LLMs approach SOTA performance on the WebNLG dataset.
  • Small Models Benefit Greatly: Smaller LLMs, like GPT-2, can achieve competitive performance with larger models when provided with KB context.
  • NYT Dataset Challenge: The NYT dataset remains challenging for LLMs, even with KB augmentation, likely due to its complex sentence structures and implicit relations.
  • KB Context Quality Matters: The study showed a strong correlation between the probability of finding the correct triplet within the retrieved KB context and the final TE performance.

Limitations and Future Work

  • Dataset Generality: The authors note that the significant overlap between train, validation, and test sets in the datasets used might inflate performance metrics and calls for revisions with better test isolation.
  • LLM Re-elaboration: While LLMs can identify relevant information in the KB context, they may not yet excel at re-elaborating and generalizing this information for different examples.
  • KB Completeness: The intrinsic incompleteness of KBs can be a limiting factor for TE performance.
  • Future Work: The paper suggests that improving the quality of the KB and its associated information retriever could be more effective than solely increasing LLM size for TE tasks. Further research into how LLMs re-elaborate retrieved information is also warranted.

Summary updated

BibTeX
@inproceedings{papaluca2024zero,
  author = {Andrea Papaluca and Daniel Krefl and Sergio Rodríguez Méndez and Artem Lensky and Hanna Suominen},
  title = {{Zero- and Few-Shots Knowledge Graph Triplet Extraction with Large Language Models}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
}
Figure