Generation-Augmented Retrieval: Rethinking the Role of Large Language Models in Zero-Shot Relation Extraction

Zehan Li, Fu Zhang, Tianyue Peng, He Liu, Jingwei ChengEMNLP 2025

TL;DR

This paper introduces RE-GAR-AD, a novel framework that reframes Zero-Shot Relation Extraction (ZSRE) as a retrieval problem, leveraging Large Language Models (LLMs) for generation-augmented embeddings. By using LLMs to enrich sentence and relation representations and employing a lightweight retrieval adjuster, RE-GAR-AD achieves state-of-the-art performance with significantly reduced computational costs compared to existing LLM-based methods.

What Problem Does This Paper Address?

Relation Extraction (RE) aims to identify the semantic relationship between two entities within a sentence from a predefined set of relations. Traditional supervised RE requires extensive labeled data, making it impractical for scenarios with novel or unseen relations. Zero-Shot Relation Extraction (ZSRE) tackles this by attempting to identify relations without any labeled examples for them. While Large Language Models (LLMs) have shown promise in various NLP tasks, their application to ZSRE often involves complex prompting strategies, manual interventions, and substantial computational overhead, especially for large-scale datasets with many candidate relations. Existing LLM-based ZSRE methods like QA4RE and SUMASK, while effective, still face challenges with input token efficiency and robustness. This paper addresses the need for a more efficient, accurate, and scalable LLM-based ZSRE approach.

Methods

The core of RE-GAR-AD is a shift from traditional classification to a retrieval-based paradigm. The framework consists of three key components:

  1. Generation-Augmented Embeddings: This module uses LLMs to enhance the semantic representation of both sentences and relation labels.
  2. Sentence Distillation Prompt: This prompt guides LLMs to extract the most relevant semantic information from a sentence, focusing on the target entities and their relationship, while filtering out noise. It also infers entity types.
  3. Relation Enrichment Prompt: This prompt leverages LLMs' general knowledge to generate a comprehensive set of possible entity types associated with a given relation label, alleviating the need for manual annotation.
  4. These enriched representations are then embedded into a unified vector space using an "Embedding LLM" (specifically, OpenAI's text-embedding-3-small).
  5. Retrieval Adjuster (AD): To address potential anisotropy in LLM embeddings (where embeddings cluster too closely), a lightweight retrieval adjuster is introduced. This module is trained on seen relation data to learn how to align sentence and relation embeddings. Crucially, it operates with only 2 million parameters and does not require access to the LLM's parameters, making it efficient and adaptable, especially for API-based LLMs. It uses a residual connection to integrate learned features with original embeddings.
  6. Triplet-based Retrieval: The framework then performs retrieval by matching the embedded sentence triplets (sentence, head entity type, tail entity type) with embedded relation triplets (relation label, head entity types, tail entity types). A similarity score is computed, and the relation with the highest score is selected.

Datasets

The proposed RE-GAR-AD framework was evaluated on several standard ZSRE benchmarks:

  • Wiki-ZSL: A large-scale dataset with imbalanced samples and noisy labels due to distant supervision.
  • FewRel: A meticulously annotated dataset with balanced samples.
  • TACRED, TACREV, and Re-TACRED: These datasets were used for evaluation when entity type information was available, allowing for filtering of candidate relations.

The experiments were conducted with varying numbers of unseen relations ($m=5, 10, 15$).

Key Insights and Ideas

  • Reframing ZSRE as Retrieval: The most significant insight is redefining ZSRE as an embedding-based retrieval task. This shift simplifies the problem, reduces token input requirements, and avoids the output format issues common in generative approaches.
  • LLMs for Semantic Enrichment: The paper demonstrates that LLMs, through carefully crafted prompts, can effectively distill crucial sentence semantics and enrich relation labels with diverse entity types, significantly improving the quality of embeddings for retrieval.
  • Lightweight, Plug-in Retrieval Adjuster: The introduction of a small, parameter-efficient retrieval adjuster that can be "plugged in" without fine-tuning the LLM itself is a key innovation. This addresses the practical challenges of adapting LLMs for ZSRE, especially with API-based models.
  • Efficiency Gains: By encoding sentences and relation labels separately and performing retrieval, the framework drastically reduces the token input overhead from $n \times k \times r$ (for $n$ sentences, $k$ responses, $r$ relations) to $n+r$, making it highly scalable.

Key Results

  • State-of-the-Art Performance: RE-GAR-AD achieves competitive performance across multiple ZSRE benchmarks. Without fine-tuning on seen data (RE-GAR), it outperforms QA4RE by an average F1 of 6.93. When the retrieval adjuster is trained on seen data (RE-GAR-AD), it further improves performance, showing an F1 advantage of 3.71 over MICRE.
  • Efficiency: The framework significantly reduces computational costs. For example, RE-GAR demonstrates advantages in both F1 score and cost compared to QA4RE.
  • Robustness: RE-GAR-AD shows robust performance across different settings and scales, even outperforming other baselines by a large margin at $m=5$ on Wiki-ZSL, attributed to effective noise reduction via Sentence Distillation.
  • Ablation Studies: Removing Sentence Distillation led to a significant performance drop (7.74 F1), highlighting its importance. The Retrieval Adjuster also contributed a notable performance boost (3.42 F1).

Limitations and Future Work

  • Cost-Effectiveness for Small Inputs: The framework's cost-effectiveness is less evident when the number of input sentences ($n$) is small.
  • API Limitations: Complete results could not be obtained for more powerful models like GPT-4 due to API limitations.
  • Retrieval Adjuster Simplicity: The current structure of the retrieval adjuster is straightforward; further research into more sophisticated LLM-embedding-based adjusters is suggested.
  • Handling "None-of-the-Above" (NOTA) Relations: The method encounters difficulties with datasets containing a "none-of-the-above" relation, limiting its applicability in such scenarios.
  • Future Work: The authors suggest optimizing the retrieval adjuster to better handle challenging cases, particularly distinguishing similar relations and addressing co-occurring relations within sentences. Enriching relation label semantics for better distinctiveness is also proposed.

Summary updated

BibTeX
@inproceedings{li2025generation,
  author = {Zehan Li and Fu Zhang and Tianyue Peng and He Liu and Jingwei Cheng},
  title = {{Generation-Augmented Retrieval: Rethinking the Role of Large Language Models in Zero-Shot Relation Extraction}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
}
Figure