GENRA: Enhancing Zero-shot Retrieval with Rank Aggregation
Introduction
In the realm of information retrieval, "zero-shot" scenarios—where a system must find relevant documents for a query without prior training on that specific domain—remain a significant challenge. Traditional systems typically employ a two-stage "retrieve then re-rank" pipeline. However, this approach suffers from a critical bottleneck: if the initial retriever fails to capture relevant documents, the subsequent re-ranker has no opportunity to correct the error.
While Large Language Models (LLMs) have been integrated to expand queries or generate hypothetical answers (e.g., HyDE), they are often prone to hallucinations and sensitive to prompt variations. GENRA (Generative Rank Aggregation) addresses these gaps by leveraging the creative power of LLMs alongside the statistical robustness of rank aggregation.
The GENRA Framework
GENRA operates through a sophisticated three-stage pipeline that requires no task-specific fine-tuning, relying entirely on LLM inference and standard retrieval techniques.
- Passage Generation: The process begins by using an LLM to generate multiple diverse passages based on the initial query. These passages represent different facets of the user's intent and are used to perform an initial search across the document corpus.
- Relevance Assessment: To filter out noise and potential LLM hallucinations, the system uses an LLM as a "judge." It performs either binary relevance checks or list-wise re-ranking on the top results from the first stage, retaining only the most relevant documents.
- Rank Aggregation: This is the core innovation. For each verified document, GENRA performs a new retrieval pass, treating the document itself as a query. This generates multiple independent ranked lists. These lists are then merged using Linear Aggregation, which sums normalized similarity scores to produce a final, high-consensus ranking.
Experimental Evaluation
The authors evaluated GENRA across a broad spectrum of benchmarks, including TREC Deep Learning (DL19, DL20), the BEIR benchmark (covering medical, news, and social media domains), and CrisisFACTS 2022.
Key Results: * Performance Gains: On TREC datasets, GENRA improved Mean Average Precision (MAP) and nDCG@10 by 0.5 to 7.4 percentage points over the competitive HyDE baseline. * Synergy with Re-rankers: When integrated with RankVicuna, GENRA achieved improvements of up to 9.9 percentage points in MAP and Recall@100. * Domain Generalization: GENRA consistently outperformed baselines across diverse BEIR datasets, proving its effectiveness in specialized fields like medicine and argumentation.
Key Insights
- Diversity over Precision: The success of GENRA suggests that aggregating results from multiple verified relevant documents provides a more comprehensive understanding of query intent than any single-path search.
- The Power of Simplicity: Linear aggregation (summing scores) was found to be more effective than complex positional methods like Borda count, primarily because it preserves the confidence levels of the underlying retrievers.
- Optimal Context: The study identified a "Goldilocks" zone, finding that generating approximately 10 passages and aggregating 5 verified documents yields the best performance balance.
Conclusion and Future Work
GENRA represents a significant step forward in zero-shot retrieval by combining LLM-driven generation with robust rank aggregation. While the pipeline introduces higher computational costs due to multiple retrieval passes, it offers a high-accuracy alternative to existing methods. Future research will focus on improving the efficiency of the aggregation step and exploring more granular relevance scoring to further refine the final rankings.
Summary updated