KeyB2: Selecting Key Blocks is Also Important for Long Document Ranking with Large Language Models.

Minghan Li, Éric Gaussier, Juntao Li, Guodong ZhouarXiv 2024 · OpenReview

Abstract

Large Language Models (LLMs) have significantly advanced document reranking, yet they struggle with long documents due to quadratic computational costs and "signal dilution" from irrelevant text. We propose EviRerank, a framework that builds a compact, evidence-focused context for LLMs. By employing Adaptive Evidence Budgeting (AEB) to filter noise and Summary Augmentation (SA) to preserve global context, EviRerank reduces inference latency by up to 80% and memory usage by over 75% while achieving state-of-the-art accuracy.

Introduction

While decoder-only LLMs like Llama excel at semantic understanding, their application to long-document reranking is hindered by two primary issues. First, the quadratic cost of self-attention makes processing long sequences computationally prohibitive. Second, the "dilution effect" occurs when sparse relevant evidence is buried within irrelevant text, causing the model's attention to disperse and weakening the signal. Existing methods often rely on simple truncation or fixed-length block selection, which fail to adapt to varying information densities across documents.

The EviRerank Framework

EviRerank addresses these challenges through a multi-stage pipeline designed to extract and organize query-relevant information efficiently.

Architecture of the EviRerank framework featuring Adaptive Evidence Budgeting (AEB) and Summary Augmentation (SA).

The process consists of four key components:

  1. Local Block Scoring: The document is partitioned into small blocks (~63 tokens). A lightweight selector (e.g., BM25, Bi-Encoder, or Cross-Encoder) assigns a relevance score to each block relative to the query.
  2. Adaptive Evidence Budgeting (AEB): Unlike fixed-budget methods, AEB uses a ratio-based stopping criterion. Blocks are sorted by score, and the selection process terminates when the marginal utility of the next block falls below a threshold ($\rho$). This allows the input length to adapt dynamically to the document's content.
  3. Summary Augmentation (SA): To provide the LLM with a global perspective, a compact, query-agnostic summary is included. This is generated by selecting blocks closest to the document's centroid embedding.
  4. LLM Reranking: The synthesized context (selected blocks and summary) is processed by a fine-tuned RankLLaMA model to generate the final relevance score.

Experimental Setup

The framework was evaluated on three diverse long-document benchmarks:

  • TREC DL 2019: Web documents from MS MARCO v1 with human judgments.
  • TREC DL 2023: Longer web-style documents from MS MARCO v2.
  • MLDR-zh: A Chinese multi-long-document retrieval benchmark with documents averaging 13,000 tokens.

Results and Discussion

EviRerank demonstrates a significant improvement in both effectiveness and efficiency over full-document baselines.

Effectiveness (NDCG@10) vs. Latency on TREC DL'19, comparing EviRerank variants against RankLLaMA baselines.

Effectiveness

On TREC DL’19, EviRerank achieved an NDCG@10 of 0.743, surpassing the full-document RankLLaMA (0.701). This confirms that removing irrelevant text can actually improve model performance by mitigating signal dilution.

Efficiency

The framework reduced reranking latency from approximately 50 seconds to 9–11 seconds per 100 documents. Furthermore, memory requirements dropped from over 96GB VRAM to roughly 22–23GB, enabling high-performance reranking on standard enterprise GPUs.

Conclusion and Future Work

EviRerank proves that "less is more" in long-document LLM reranking. By intelligently selecting evidence and augmenting it with global summaries, we achieve superior accuracy with a fraction of the computational cost. Future research will explore more sophisticated, query-aware summary modules and the generalizability of these techniques across different LLM architectures.

Summary updated

BibTeX
@misc{li2024keyb,
  author = {Minghan Li and Éric Gaussier and Juntao Li and Guodong Zhou},
  title = {{KeyB2: Selecting Key Blocks is Also Important for Long Document Ranking with Large Language Models.}},
  year = {2024},
  url = {https://openreview.net/forum?id=iFN3lmnQDZ},
}
Figure