TL;DR
This paper introduces DeAR, a novel dual-stage framework for document reranking that effectively distills knowledge from large language models (LLMs) into smaller, more efficient models. By combining pointwise relevance scoring with listwise reasoning augmented by chain-of-thought (CoT) explanations, DeAR achieves state-of-the-art performance on various benchmarks while remaining lightweight and interpretable.
What Problem Does This Paper Address?
Document reranking is crucial for improving search results, question answering, and other information retrieval tasks. While large LLMs have shown promise in global reasoning over candidate documents, they often struggle with balancing fine-grained relevance scoring with holistic cross-document analysis. Furthermore, relying solely on large, proprietary LLMs presents challenges related to cost, context length limitations, and brittle reasoning. Existing methods for distilling LLM knowledge into smaller models often fail to capture the full spectrum of reasoning capabilities, leading to performance degradation. This paper aims to bridge this gap by developing an efficient, open-source reranking framework that leverages LLM distillation for both pointwise scoring and listwise reasoning.
Methods
DeAR employs a dual-stage approach:
- Stage 1: Pointwise Reranking with KL Distillation: This stage focuses on distilling token-level relevance signals from a large "teacher" LLM (e.g., LLaMA-13B) into a compact "student" model (e.g., 3B or 8B parameters). A hybrid loss function combining cross-entropy, RankNet, and KL divergence is used to ensure robust pointwise scoring. KL divergence helps align the student's score distribution with the teacher's, while the ranking losses guide the student to learn effective ranking strategies. Lightweight LoRA adapters are used to make the student model efficient.
- Stage 2: Reasoning Listwise Reranking: In this stage, a second LoRA adapter is attached to the student model, and it's fine-tuned on synthetic "chain-of-thought" (CoT) permutations generated by a powerful LLM (GPT-4o). These CoT examples provide step-by-step reasoning and justifications for the rankings, enabling the student model to perform listwise reasoning and capture inter-document dependencies. This stage enhances the global ranking quality and interpretability.
Datasets
The authors evaluated DeAR on several standard benchmarks:
- TREC Deep Learning Tracks (DL19, DL20): These are established datasets for evaluating document retrieval and reranking systems.
- BEIR (Benchmarking Information Retrieval): A collection of diverse information retrieval datasets, including Covid, NFCorpus, Signal, News, Robust04, Touche, DBPedia, and SciFact.
- NovelEval-2306: A benchmark for evaluating generalization on unseen queries.
- Natural Questions (NQ) and Web Questions (WebQ): Datasets used for open-domain question answering.
The synthetic data for Stage 2 was generated from MS MARCO queries.
Key Insights and Ideas
- Decoupling for Efficiency and Performance: Separating pointwise scoring from listwise reasoning allows for specialized optimization of each task, leading to better overall performance and interpretability.
- Hybrid Distillation for Robustness: Combining cross-entropy, RankNet, and KL divergence losses in Stage 1 effectively transfers nuanced relevance judgments from a teacher LLM to a student model, ensuring stable calibration.
- Chain-of-Thought for Listwise Reasoning: Using GPT-4o-generated CoT examples in Stage 2 significantly enhances the student model's ability to perform listwise reasoning, capturing complex inter-document relationships that pointwise methods miss.
- Open-Source Efficiency: DeAR demonstrates that high-performance reranking can be achieved with compact, open-source models, reducing reliance on expensive proprietary APIs.
- Interpretability through Reasoning: The CoT explanations generated in Stage 2 not only improve ranking quality but also provide insights into why certain documents are ranked higher, making the system more interpretable.
Key Results
- Superior Performance: DeAR-L (listwise) achieved 90.97 nDCG@10 on NovelEval-2306, outperforming GPT-4 by +3.09. On TREC DL20, it surpassed open-source baselines by +5.1 nDCG@5.
- Competitive in Open-Domain QA: DeAR achieved 54.29 Top-1 accuracy on Natural Questions, outperforming baselines like MonoT5 and RankGPT.
- Efficiency: The pointwise stage (DeAR-P) achieved 74.5 nDCG@10 in 2.2 seconds on TREC DL19, significantly faster than baselines like RankZephyr (21.58 seconds). The listwise stage (DeAR-L) achieved 75.54 nDCG@10 in 11.16 seconds.
- Scalability: Even smaller student models (e.g., LLaMA3.1-3B) showed competitive performance, underscoring DeAR's efficiency.
- Teacher-Student Flexibility: DeAR demonstrated adaptability across various teacher and student model pairs, with LLaMA2-13B to LLaMA3.1-8B yielding the highest BEIR average.
Limitations and Future Work
- Reliance on Synthetic Data: The listwise reasoning stage relies on synthetic data generated by GPT-4o. This could potentially introduce biases or errors from the teacher model, affecting generalization.
- Dependency on Initial Retrieval: DeAR's performance is evaluated on top-100 candidates retrieved by BM25, meaning its effectiveness is inherently tied to the quality of the initial retrieval stage.
- Context Window Constraints: The listwise stage processes smaller candidate sets (e.g., top-20), which might limit its ability to handle larger sets due to LLM context window limitations.
- Computational Complexity: While efficient compared to some baselines, the two-stage process of DeAR increases computational complexity compared to single-stage rerankers, which could be a challenge in resource-constrained environments.
The authors suggest that future work could explore more robust methods for synthetic data generation and investigate how DeAR can be adapted to handle larger candidate sets.
Summary updated