Abstract
Large Language Models (LLMs) have significantly advanced document reranking, yet they struggle with long documents due to quadratic computational costs and "signal dilution" from irrelevant text. We propose EviRerank, a framework that builds a compact, evidence-focused context for LLMs. By employing Adaptive Evidence Budgeting (AEB) to filter noise and Summary Augmentation (SA) to preserve global context, EviRerank reduces inference latency by up to 80% and memory usage by over 75% while achieving state-of-the-art accuracy.
Introduction
While decoder-only LLMs like Llama excel at semantic understanding, their application to long-document reranking is hindered by two primary issues. First, the quadratic cost of self-attention makes processing long sequences computationally prohibitive. Second, the "dilution effect" occurs when sparse relevant evidence is buried within irrelevant text, causing the model's attention to disperse and weakening the signal. Existing methods often rely on simple truncation or fixed-length block selection, which fail to adapt to varying information densities across documents.
The EviRerank Framework
EviRerank addresses these challenges through a multi-stage pipeline designed to extract and organize query-relevant information efficiently.
The process consists of four key components:
- Local Block Scoring: The document is partitioned into small blocks (~63 tokens). A lightweight selector (e.g., BM25, Bi-Encoder, or Cross-Encoder) assigns a relevance score to each block relative to the query.
- Adaptive Evidence Budgeting (AEB): Unlike fixed-budget methods, AEB uses a ratio-based stopping criterion. Blocks are sorted by score, and the selection process terminates when the marginal utility of the next block falls below a threshold ($\rho$). This allows the input length to adapt dynamically to the document's content.
- Summary Augmentation (SA): To provide the LLM with a global perspective, a compact, query-agnostic summary is included. This is generated by selecting blocks closest to the document's centroid embedding.
- LLM Reranking: The synthesized context (selected blocks and summary) is processed by a fine-tuned RankLLaMA model to generate the final relevance score.
Experimental Setup
The framework was evaluated on three diverse long-document benchmarks:
- TREC DL 2019: Web documents from MS MARCO v1 with human judgments.
- TREC DL 2023: Longer web-style documents from MS MARCO v2.
- MLDR-zh: A Chinese multi-long-document retrieval benchmark with documents averaging 13,000 tokens.
Results and Discussion
EviRerank demonstrates a significant improvement in both effectiveness and efficiency over full-document baselines.
Effectiveness
On TREC DL’19, EviRerank achieved an NDCG@10 of 0.743, surpassing the full-document RankLLaMA (0.701). This confirms that removing irrelevant text can actually improve model performance by mitigating signal dilution.
Efficiency
The framework reduced reranking latency from approximately 50 seconds to 9–11 seconds per 100 documents. Furthermore, memory requirements dropped from over 96GB VRAM to roughly 22–23GB, enabling high-performance reranking on standard enterprise GPUs.
Conclusion and Future Work
EviRerank proves that "less is more" in long-document LLM reranking. By intelligently selecting evidence and augmenting it with global summaries, we achieve superior accuracy with a fraction of the computational cost. Future research will explore more sophisticated, query-aware summary modules and the generalizability of these techniques across different LLM architectures.
Summary updated