TL;DR
This paper introduces MVP, a novel passage reranking model that significantly improves efficiency and robustness by using multiple "views" of query-passage relationships and a non-generative, single-pass decoding approach. MVP achieves state-of-the-art performance comparable to much larger models while drastically reducing inference latency, making it more practical for real-world applications.
What Problem Does This Paper Address?
Passage reranking is a crucial step in information retrieval, where a system refines a list of candidate passages (often retrieved by a simpler method) to present the most relevant ones to a user's query. Large Language Models (LLMs) have shown great promise in this area, but existing LLM-based methods face two major hurdles:
- Efficiency: Many LLM rerankers rely on autoregressive generation or sliding windows to process passages. This leads to high computational costs, especially when dealing with a large number of candidate passages, making real-time application challenging.
- Robustness to Biases: LLM rerankers can be sensitive to external biases. "Position bias" occurs when the model's performance is affected by the order of passages in the input, and "selection bias" arises when the way passages are identified (e.g., using simple tokens like "A" or "1") inadvertently influences the model's judgment. These biases hinder accurate passage representation.
Methods
The authors propose Multi-View-guided Passage Reranking (MVP), a non-generative LLM-based reranking model. Key components include:
- Multi-View Encoding: Instead of a single representation, MVP encodes each query-passage pair into multiple "relevance vectors," each representing a distinct "view" of the relationship. This is achieved using a Fusion-in-Decoder (FiD) architecture with learnable soft prompts. Crucially, these prompts are inserted at fixed positions for all passages, and distinct position embeddings are used for each view, mitigating position bias.
- Anchor-Guided Decoding: MVP employs a non-generative decoding strategy. For each view, an "anchor vector" is generated. The relevance score for a passage is then computed by measuring the similarity between its relevance vectors and the anchor vectors. This allows for direct scoring and global ranking in a single decoding step, eliminating the need for generating passage identifiers sequentially.
- Orthogonal Loss: To ensure that each view captures a distinct aspect of relevance and to encourage diversity among the anchor vectors, an orthogonality regularization term is added to the training objective.
Datasets
- Training: The model was trained on the Rank-DistiLLM dataset, derived from MS MARCO passage ranking data.
- Evaluation:
- In-domain: TREC-DL19, DL20.
- Out-of-domain (Zero-shot): BEIR benchmark (comprising datasets like Covid, NFCorpus, Signal, News, Robust04, SciFact, Touche, DBPedia).
Key Insights and Ideas
- Decoupling Relevance from Generation: MVP demonstrates that effective reranking doesn't require generating passage identifiers. By directly computing similarity scores between relevance vectors and anchors, it bypasses the inefficiencies of generative models.
- Robustness through Fixed Views: The fixed positional embedding of view tokens across all passages, regardless of the candidate order, is key to MVP's resilience against position bias.
- Bias Mitigation via Soft Prompts: Using learnable soft prompts for views, rather than natural language tokens as identifiers, effectively sidesteps selection bias.
- Efficiency through Single-Pass Global Ranking: The anchor-guided decoding allows all candidate passages to be evaluated simultaneously in one pass, drastically reducing computational overhead.
Key Results
- Performance: MVP (220M parameters) matches or surpasses the performance of much larger 7B-scale fine-tuned models on TREC-DL19 and DL20. The MVP-3B variant achieves state-of-the-art results on both in-domain and out-of-domain benchmarks.
- Efficiency: MVP achieves up to a 100x reduction in inference latency compared to larger models. It also significantly reduces FLOPs, using approximately 82% fewer operations than ListT5.
- Robustness: MVP shows consistent performance across different candidate passage orders (position bias) and is not affected by identifier reordering (selection bias), unlike many other listwise rerankers.
Limitations and Future Work
- Fixed Number of Views: MVP uses a fixed number of views across all datasets. The authors suggest that exploring dynamic view selection or learning query-specific view weights could further improve performance.
- Uniform View Aggregation: Currently, all views are aggregated with equal weights. Investigating dynamic weighting based on query characteristics could be a future direction.
Summary updated