Abstract
Information Retrieval (IR) for long documents faces a persistent challenge: bi-encoders often suffer from signal dilution ("lost in the middle"), while cross-encoders are computationally prohibitive for long sequences. This paper introduces BReps, a framework that segments long documents into coherent blocks and encodes them into individual embeddings. By integrating these with a lightweight Top-k Interaction Refinement (TIR) module, we achieve state-of-the-art accuracy on long-document benchmarks. BReps-TIR demonstrates superior efficiency, being significantly faster than traditional interaction-based models while maintaining high precision.
Introduction
The "Goldilocks" problem in IR describes the tension between speed and accuracy. Bi-encoders compress entire documents into single vectors, which is fast but often loses local evidence. Conversely, cross-encoders process query-document pairs jointly, capturing fine-grained interactions at the cost of quadratic scaling in self-attention. We propose a middle ground: representing documents as sets of block-level embeddings (BReps) and refining their relevance scores through a small, efficient interaction module (TIR).
Methodology
The BReps framework operates in two stages: offline block encoding and online interaction refinement.
Block Representations (BReps)
Documents are segmented into semantically coherent blocks (up to 63 tokens) using CogLTX, a dynamic programming-based approach. Each block is encoded offline using a decoder-only Large Language Model (e.g., Gemma-2-2B). This decomposition reduces offline encoding time by approximately 67.9% compared to processing full sequences, as the quadratic cost of attention is minimized across smaller segments.
Top-k Interaction Refinement (TIR)
At query time, the system retrieves the top-$k$ most relevant blocks based on initial similarity scores. To account for dependencies between blocks—such as redundancy or complementary information—we introduce the TIR module.
TIR is a 3-million parameter module that performs setwise attention over the top-$k$ blocks (e.g., $k=65$). It produces a residual correction ("delta") to the initial scores, effectively calibrating the final ranking without the heavy overhead of a full cross-encoder.
Experimental Evaluation
We evaluated BReps-TIR on several long-document benchmarks, including TREC Deep Learning (DL) 2019, 2022, 2023, and the multilingual MLDR-zh dataset.
Key Results
BReps-TIR achieves a superior balance of effectiveness and efficiency. On the TREC DL’19 benchmark, it reached an NDCG@10 of 0.700, surpassing both the single-vector RepLLaMA baseline (0.657) and the interaction-based IDCM (0.681).
As shown in the Pareto frontier, BReps-TIR provides a significant speedup, reranking 100 candidates in roughly 290ms on an NVIDIA A100, compared to over 1200ms for IDCM.
Discussion and Insights
- Data Efficiency: The model was trained on only 20,000 triplets, indicating that block-level supervision is highly effective for learning relevance.
- Interpretability: By scoring individual blocks, the system provides inherent transparency, allowing users to identify specific document segments that contributed to a ranking.
- Efficiency: The transition from document-level to block-level processing not only improves retrieval quality but also reduces offline computation time significantly compared to full-sequence encoding.
Conclusion
BReps-TIR offers a scalable solution for long-document retrieval by combining the strengths of fine-grained representations with lightweight interaction. Future work will explore adaptive block budgeting and the application of block embeddings to the initial retrieval stage.
Summary updated