Enhanced Retrieval of Long Documents: Leveraging Fine-Grained Block Representations with Large Language Models.

Minghan Li, Éric Gaussier, Guodong ZhouarXiv 2025 · OpenReview

Abstract

Information Retrieval (IR) for long documents faces a persistent challenge: bi-encoders often suffer from signal dilution ("lost in the middle"), while cross-encoders are computationally prohibitive for long sequences. This paper introduces BReps, a framework that segments long documents into coherent blocks and encodes them into individual embeddings. By integrating these with a lightweight Top-k Interaction Refinement (TIR) module, we achieve state-of-the-art accuracy on long-document benchmarks. BReps-TIR demonstrates superior efficiency, being significantly faster than traditional interaction-based models while maintaining high precision.

Introduction

The "Goldilocks" problem in IR describes the tension between speed and accuracy. Bi-encoders compress entire documents into single vectors, which is fast but often loses local evidence. Conversely, cross-encoders process query-document pairs jointly, capturing fine-grained interactions at the cost of quadratic scaling in self-attention. We propose a middle ground: representing documents as sets of block-level embeddings (BReps) and refining their relevance scores through a small, efficient interaction module (TIR).

Methodology

The BReps framework operates in two stages: offline block encoding and online interaction refinement.

Block Representations (BReps)

Documents are segmented into semantically coherent blocks (up to 63 tokens) using CogLTX, a dynamic programming-based approach. Each block is encoded offline using a decoder-only Large Language Model (e.g., Gemma-2-2B). This decomposition reduces offline encoding time by approximately 67.9% compared to processing full sequences, as the quadratic cost of attention is minimized across smaller segments.

Top-k Interaction Refinement (TIR)

At query time, the system retrieves the top-$k$ most relevant blocks based on initial similarity scores. To account for dependencies between blocks—such as redundancy or complementary information—we introduce the TIR module.

The architecture of the TIR module, featuring Attention Context, Additive Fusion, and Score Gate components for refining block-level relevance.

TIR is a 3-million parameter module that performs setwise attention over the top-$k$ blocks (e.g., $k=65$). It produces a residual correction ("delta") to the initial scores, effectively calibrating the final ranking without the heavy overhead of a full cross-encoder.

Experimental Evaluation

We evaluated BReps-TIR on several long-document benchmarks, including TREC Deep Learning (DL) 2019, 2022, 2023, and the multilingual MLDR-zh dataset.

Key Results

BReps-TIR achieves a superior balance of effectiveness and efficiency. On the TREC DL’19 benchmark, it reached an NDCG@10 of 0.700, surpassing both the single-vector RepLLaMA baseline (0.657) and the interaction-based IDCM (0.681).

Pareto frontier comparing NDCG@10 vs. online latency, showing BReps-TIR achieving higher accuracy at significantly lower latency than IDCM and KeyB.

As shown in the Pareto frontier, BReps-TIR provides a significant speedup, reranking 100 candidates in roughly 290ms on an NVIDIA A100, compared to over 1200ms for IDCM.

Discussion and Insights

  • Data Efficiency: The model was trained on only 20,000 triplets, indicating that block-level supervision is highly effective for learning relevance.
  • Interpretability: By scoring individual blocks, the system provides inherent transparency, allowing users to identify specific document segments that contributed to a ranking.
  • Efficiency: The transition from document-level to block-level processing not only improves retrieval quality but also reduces offline computation time significantly compared to full-sequence encoding.

Conclusion

BReps-TIR offers a scalable solution for long-document retrieval by combining the strengths of fine-grained representations with lightweight interaction. Future work will explore adaptive block budgeting and the application of block embeddings to the initial retrieval stage.

Summary updated

BibTeX
@misc{li2025enhanced,
  author = {Minghan Li and Éric Gaussier and Guodong Zhou},
  title = {{Enhanced Retrieval of Long Documents: Leveraging Fine-Grained Block Representations with Large Language Models.}},
  year = {2025},
  url = {https://openreview.net/forum?id=RrOmcosuFG},
}

Loading related content...

Figure