Referral Augmentation for Zero-Shot Information Retrieval

Michael Tang, Shunyu Yao, John Yang, Karthik Narasimhan

Abstract

We propose Referral-Augmented Retrieval (RAR), a simple technique that concatenates document indices with referrals—text from other documents that cite or link to the given document—to provide significant performance gains for zero-shot information retrieval. The key insight is that referrals provide a multi-view representation of a document, similar to how incoming links function in PageRank. RAR works with both sparse and dense retrievers and outperforms generative text expansion techniques like DocT5Query and Query2Doc, achieving up to a 37% absolute improvement on ACL paper retrieval Recall@10 while eliminating expensive model training.

Introduction

In zero-shot information retrieval (IR), systems must find relevant documents in new domains without prior exposure to domain-specific query-document pairs. A primary challenge is the lexical gap, where user queries use different terminology than the target document's author. While generative "document expansion" methods attempt to bridge this gap by hallucinating potential queries, they are computationally expensive and prone to noise.

Referral-Augmented Retrieval (RAR) addresses this by leveraging existing human-authored descriptions: citations and hyperlinks. These "referrals" act as naturally occurring "hard positives," providing diverse perspectives on a document's content that the original author might have omitted.

Methods

The RAR framework enhances the search index by incorporating referral text—the context surrounding a citation or link in a source document—into the target document's representation.

Overview of Referral-Augmented Retrieval (RAR) compared to standard retrieval.

The authors explored three integration strategies:

  1. Concatenation: Appending referral text to the document, which proved most effective for sparse models like BM25.
  2. Mean Representation: Averaging vector embeddings of the document and its referrals, optimal for dense models like SimCSE.
  3. Shortest Path: Treating each referral as a separate entry to find the closest match to a query, enhancing high-precision retrieval.

RAR is entirely training-free, requiring only a modification to the indexing stage rather than fine-tuning underlying transformer models.

Experimental Setup

RAR was evaluated across two primary domains:

  • Scientific Paper Retrieval: Using the S2ORC (ACL and ArXiv subsets) and RefSeer corpora. The task involved local citation recommendation—identifying the correct paper for a masked citation.
  • Entity Retrieval: Using DBPedia (Wikipedia). The task was to retrieve the correct entity page based on free-text queries, using internal Wikipedia hyperlinks as referrals.

Results

The implementation of RAR yielded substantial performance improvements across all tested configurations.

Performance gains in Recall@10 across three datasets using RAR with sparse and dense retrievers.

Key findings include:

  • Sparse Retriever Gains: On the ACL dataset, adding referrals to BM25 increased Recall@10 from 26.5% to 50.5%, a nearly 100% relative improvement.
  • Outperforming Generative Baselines: RAR surpassed state-of-the-art generative expansion techniques like DocT5Query and Query2Doc by 37% and 21% absolute Recall@10, respectively.
  • Cross-Model Versatility: The technique consistently improved performance for both sparse (BM25) and dense (SimCSE, DPR) retrievers.

Discussion and Conclusion

RAR demonstrates that the "multi-view" perspective provided by external citations significantly enhances document findability. Unlike generative methods, RAR provides "gold-quality" human-curated data and allows for dynamic index updates without retraining.

However, the method's effectiveness depends on the connectivity of the corpus; it is less effective for isolated documents. Additionally, large numbers of referrals can lead to index bloat and increased search latency. Future research will focus on referral selection and filtering to maintain high performance while optimizing index efficiency.

Summary updated

Abstract

We propose Referral-Augmented Retrieval (RAR), a simple technique that concatenates document indices with referrals, i.e. text from other documents that cite or link to the given document, to provide significant performance gains for zero-shot information retrieval. The key insight behind our method is that referrals provide a more complete, multi-view representation of a document, much like incoming page links in algorithms like PageRank provide a comprehensive idea of a webpage's importance. RAR works with both sparse and dense retrievers, and outperforms generative text expansion techniques such as DocT5Query and Query2Doc a 37% and 21% absolute improvement on ACL paper retrieval Recall@10 -- while also eliminating expensive model training and inference. We also analyze different methods for multi-referral aggregation and show that RAR enables up-to-date information retrieval without re-training.
BibTeX
@inproceedings{tang2024referral,
  author = {Michael Tang and Shunyu Yao and John Yang and Karthik Narasimhan},
  title = {{Referral Augmentation for Zero-Shot Information Retrieval}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
  url = {https://openreview.net/forum?id=cZkXrsORYR},
}
Figure