MuGI: Enhancing Information Retrieval through Multi-Text Generation Intergration with Large Language Models

Le Zhang, Yihong Wu

Abstract

Large Language Models (LLMs) have become foundational in information retrieval (IR), particularly for query expansion. This paper introduces Multi-Text Generation Integration (MuGI), a training-free framework that leverages LLMs to generate multiple pseudo-references. By integrating these references with user queries, MuGI enhances both sparse and dense retrievers. Our findings demonstrate that increasing LLM samples improves performance, and effective integration strategies are critical. Notably, MuGI enables a 23M-parameter model to outperform a 7B-parameter baseline, highlighting the power of contextual information in query representation.

Introduction: Solving the Vocabulary Mismatch

In Information Retrieval, the "vocabulary mismatch" problem occurs when a user's query terms do not align with the specific wording of relevant documents. While query expansion using LLMs (like the HyDE or query2doc methods) has shown promise, existing approaches often struggle to balance generated text with original intent, maintain compatibility across different retriever types (sparse vs. dense), and manage the strict token limits of neural models.

The MuGI Framework

MuGI addresses these challenges through a two-stage "Retrieve & Rerank" pipeline designed to be training-free and versatile.

The MuGI framework architecture, illustrating the two-stage process of Retrieval and Reranking using pseudo-references and neural calibration.
  1. Multi-Text Generation: MuGI generates multiple (typically five) pseudo-passages using zero-shot prompting to capture a broader context.
  2. Adaptive Reweighting for Sparse Retrieval: To ensure the original query isn't overshadowed in BM25 searches, MuGI uses an adaptive factor ($\lambda$) to repeat the original query based on the length of the generated passages.
  3. Contextualized Pooling for Dense Retrieval: To bypass token limits, MuGI encodes each (Query + Passage) pair separately and averages the embeddings into a single contextualized vector.
  4. Neural Calibration: Inspired by the Rocchio algorithm, MuGI refines the query vector by shifting it toward generated passages and away from potential "hard negatives."

Key Insights and Results

Evaluation on TREC DL and BEIR benchmarks revealed several critical insights into the effectiveness of query expansion:

  • Quantity and Quality: Increasing the number of LLM-generated samples consistently improves retrieval performance, though benefits typically plateau after five passages. Furthermore, the choice of LLM matters; ChatGPT-4 consistently provides superior expansions compared to smaller or older models.
  • Small Models, Big Impact: A striking result is that a tiny 23M-parameter model (MiniLM) using MuGI can outperform a 7B-parameter baseline (E5-Mistral). This suggests that high-quality query representation can compensate for smaller model sizes.
  • Broad Applicability: MuGI achieved significant gains across the board, improving BM25 performance by up to 19.8% on TREC DL and boosting various dense bi-encoders by over 7%.
Performance analysis showing how the number of generated passages and the choice of LLM (ChatGPT-4 vs. others) impact retrieval accuracy.

Conclusion

MuGI demonstrates that multi-text integration is a powerful, versatile tool for modern IR systems. While the primary bottleneck remains the latency associated with generating multiple passages from high-end LLMs, the framework sets a new standard for query expansion quality. Future work will focus on optimizing this process, perhaps through distilled LLMs or more efficient Retrieval-Augmented Generation (RAG) architectures.

Summary updated

Abstract

Large Language Models (LLMs) are foundational in language technologies, particularly in information retrieval (IR). Previous studies have utilized LLMs for query expansion, achieving notable improvements in IR. In this paper, we thoroughly explore the best practice of leveraging LLMs for query expansion. To this end, we introduce a training-free, straightforward yet effective framework called Multi-Text Generation Integration (\textsc{MuGI}). It leverages LLMs to generate multiple pseudo-references, integrating them with queries to enhance both sparse and dense retrievers. Our empirical findings reveal that: (1) Increasing the number of samples from LLMs benefits IR systems; (2) A balance between the query and pseudo-documents, and an effective integration strategy, is critical for high performance; (3) Contextual information from LLMs is essential, even boost a 23M model to outperform a 7B baseline model; (4) Pseudo relevance feedback can further calibrate queries for improved performance; and (5) Query expansion is widely applicable and versatile, consistently enhancing models ranging from 23M to 7B parameters. Our code and all generated references are made available at \url{https://github.com/lezhang7/Retrieval_MuGI}
BibTeX
@inproceedings{zhang2024mugi,
  author = {Le Zhang and Yihong Wu},
  title = {{MuGI: Enhancing Information Retrieval through Multi-Text Generation Intergration with Large Language Models}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2024},
  url = {https://openreview.net/forum?id=3bfZpWmJPO},
}
Figure