Abstract
Dense encoders and LLM-based rerankers struggle with long documents: single-vector representations dilute fine-grained relevance, while cross-encoders are often too expensive for practical reranking. We present an efficient long-document reranking framework based on block-level embeddings. Each document is segmented into short blocks and encoded into block embeddings that can be precomputed offline. Given a query, we encode it once and score each candidate document by aggregating top-k query-block similarities with a simple weighted sum, yielding a strong and interpretable block-level relevance signal. To capture dependencies among the selected blocks and suppress redundancy, we introduce Top-k Interaction Refinement (TIR), a lightweight setwise module that applies query-conditioned attention over the top-k blocks and produces a bounded residual correction to block scores. TIR introduces only a small number of parameters and operates on top-k blocks, keeping query-time overhead low. Experiments on long-document reranking benchmarks (TREC DL and MLDR-zh) show that block representations substantially improve over single-vector encoders, and TIR provides consistent additional gains over strong long-document reranking baselines while maintaining practical reranking latency. For example, on TREC DL 2023, NDCG at 10 improves from 0.395 to 0.451 with the same block budget k = 65, using at most 4095 tokens.
Introduction
In Information Retrieval (IR), a fundamental trade-off exists between efficiency and effectiveness. Dense encoders, such as RepLLaMA, compress entire documents into single vectors. While efficient, this approach often "dilutes" fine-grained relevance signals in long documents, where relevant snippets are buried within irrelevant text. Conversely, cross-encoders offer high accuracy by processing queries and documents together but suffer from quadratic computational costs, making them impractical for real-time reranking of long sequences. Furthermore, Large Language Models (LLMs) often exhibit a "lost in the middle" phenomenon, struggling to utilize information located in the center of long context windows.
This paper introduces BReps, a framework that segments long documents into semantically coherent blocks and encodes them independently. By maintaining block-level granularity, the model preserves local evidence. To further enhance performance without sacrificing speed, we introduce Top-k Interaction Refinement (TIR), a lightweight module that calibrates relevance scores by considering interactions between the most relevant blocks.
Methodology
BReps: Block-level Representations
The BReps framework begins with semantic segmentation. Instead of fixed-length chunks, documents are split into coherent blocks (maximum 63 tokens) using a dynamic programming approach (CogLTX) based on punctuation. Each block is encoded offline into a vector using a Gemma2-2B model fine-tuned with LoRA. This offline processing avoids the quadratic costs associated with long-sequence attention. At query time, the system calculates cosine similarity between the query and all blocks, aggregating the scores of the top-$k$ blocks via a weighted sum.
TIR: Top-k Interaction Refinement
While block-level scoring is effective, it treats blocks independently, potentially ignoring redundancies or complementary information. The TIR module, a tiny 3M-parameter component, addresses this by performing setwise attention on the top-$k$ blocks.
As shown in the architecture diagram, TIR calculates a "residual correction"—a small adjustment to the initial scores—to better calibrate the final document ranking. This module operates only on the top-$k$ blocks (e.g., $k=65$), ensuring query-time overhead remains minimal.
Experimental Results
The framework was evaluated on high-stakes long-document benchmarks, including TREC DL (2019, 2022, 2023) and MLDR-zh.
Effectiveness and Efficiency Trade-offs
BReps-TIR achieves a superior balance between ranking quality and latency. On TREC DL 2023, it improved NDCG@10 from 0.395 to 0.451 compared to the single-vector RepLLaMA baseline using the same token budget.
As illustrated in the Pareto frontier, BReps-TIR is significantly faster than other interaction-heavy rerankers like IDCM and KeyB while delivering higher accuracy. Specifically, BReps-TIR achieves its results in roughly 290ms, compared to over 1100ms for competing methods.
Computational Gains
The block-level approach also yields significant offline benefits. Pre-computing embeddings for 5,000 long documents took only 23.4 minutes with BReps, compared to 73 minutes for standard LLM encoding—a 67.9% reduction in overhead. This efficiency stems from processing short blocks rather than long, monolithic sequences, which avoids the quadratic scaling of Transformer attention.
Conclusion
BReps and TIR offer a scalable solution for long-document reranking. By shifting from single-vector compression to block-level representations and applying lightweight interaction refinement, the framework achieves state-of-the-art results with practical latency. The approach is inherently interpretable, as final scores are derived from specific, identifiable document blocks. Future work will explore extending this framework to initial retrieval stages and implementing adaptive block budgeting.
Summary updated