Lost in a Single Vector: Improving Long-Document Retrieval with Chunk Evidence Aggregation

Shanshan Lyu, Yiwei Wang, Yujun Cai, Jiafeng Guo, Shenghua LiuarXiv 2026 · arXiv

TL;DR

Standard dense retrieval models struggle with long documents because short, critical pieces of evidence get diluted when compressed into a single vector. This paper introduces DICE (Document Inference via Chunk Evidence), a training-free method that chunks long documents, encodes them independently using local position IDs, and aggregates them back into a single vector. DICE dramatically boosts long-document retrieval performance across multiple model architectures while preserving the standard, highly scalable one-query-one-document interface.

What Problem Does This Paper Address?

Dense retrieval systems typically encode an entire document into a single vector to rank it against a query vector. While modern embedding models boast massive context windows (e.g., 4k to 32k+ tokens), their retrieval performance still degrades sharply on long texts.

The authors identify the root cause as document-side early compression (or evidence dilution). When a query's answer lies within a short, localized span buried inside thousands of irrelevant tokens, encoding the entire document at once mixes the crucial signal with background noise before any query comparison happens. As a result, the gold document receives a low similarity score.

Figure 1: A motivating long-document retrieval case. The gold document contains a decisive local span, yet single-vector encoding under-ranks it because the evi- dence is diluted before retrieval.

To quantify this phenomenon, the authors introduce the Evidence Dilution Index (EDI), a diagnostic metric that measures how far a document-level vector representation falls below its strongest chunk-level evidence.

Methods

To combat evidence dilution without sacrificing the simplicity of the single-vector interface, the authors propose DICE (Document Inference via Chunk Evidence). DICE is a training-free, document-side encoding strategy consisting of three steps:

  1. Chunking: The long document is split into non-overlapping chunks of a fixed token size $k$ (defaulting to 1024 tokens).
  2. Local Position Encoding: Each chunk is encoded independently using a frozen embedding model. Crucially, the positional indices are reset to start from zero for each chunk, treating them as self-contained context windows.
  3. Aggregation: The resulting chunk embeddings are fused into a single document vector using a query-independent aggregation function $g(\cdot)$. Through empirical testing, the authors found that simple mean pooling works best, preserving evidence from all chunks equally.

Because all modifications happen purely on the document-encoding side, the query path and the standard vector-search infrastructure remain completely untouched.

Figure 2: Overview of DICE. Only the document side changes: chunks are encoded independently with local positions and aggregated into one vector. The query path and the retrieval interface remain unchanged.

Datasets

The authors evaluated DICE using two primary benchmarks:

  • LongEmbed: A benchmark specifically designed for long-context retrieval, featuring two synthetic tasks (Passkey and Needle in a haystack) and four real-world retrieval datasets (NarrativeQA, QMSum, SummScreenFD, and WikiMQA).
  • FollowIR: A benchmark used for secondary transfer evaluation to determine if the gains generalize to instruction-following retrieval tasks.

The evaluations spanned four diverse model backbones: Dream (diffusion-based), Llama3-8B, Mistral-7B, and Qwen2.5-7B (autoregressive models converted to bidirectional embedders).

Key Insights and Ideas

  • Capacity vs. Representation: The long-document retrieval bottleneck is not a context-window capacity issue, but a representational one. Models have the space for long contexts, but global compression washes out localized details.
  • Simplicity Wins: More sophisticated aggregation techniques, such as max-pooling or selecting chunks based on embedding norms (top-$k$), actually degrade performance on real-world tasks. Conservative mean pooling is the most robust strategy because it preserves distributed evidence without over-indexing on noisy peaks.
  • Local Positions are Crucial: Resetting position encodings for each chunk outperforms keeping absolute document offsets. Local resets ensure chunks are processed with high fidelity, whereas absolute offsets can push tokens into ranges where the model's positional embeddings are less robust or out-of-bounds.

Key Results

DICE delivers massive performance leaps, particularly on the longest and most challenging document slices:

  • Synthetic Tasks (>4k tokens): For the Dream backbone, Passkey >4k accuracy soared from 30.0% to 90.0%, and Needle >4k accuracy jumped from 23.3% to 74.0%.
  • Mitigating Length Degradation: As context length scales up to 32k tokens, standard single-vector encoding performance collapses toward zero, while DICE maintains stable, high retrieval accuracy.
Figure 3: Needle Hit@1 (%) by context length on Dream. Chunk-level encoding prevents the sharp long- context degradation observed under single-vector docu- ment compression.
  • EDI Validation: Across 12,779 evaluated samples, DICE successfully achieved a lower Evidence Dilution Index than the single-vector baseline in 92.8% of cases, proving mathematically that it keeps the final document vector closer to the actual evidence.

Limitations and Future Work

While DICE provides a powerful, training-free fix, it comes with a few trade-offs:

  • Computational Cost: Encoding a document chunk-by-chunk increases document-side compute time by roughly 3× to 4×. This trade-off is highly acceptable for offline indexing (where documents are encoded once and searched many times) but less ideal for real-time document streaming.
  • Query-Blind Aggregation: Because aggregation happens before the query is seen, DICE cannot dynamically weight chunks that are highly relevant to a specific user prompt.

The authors note that exploring learned, query-independent compression techniques to pack multi-vector chunk representations into a single vector remains an exciting direction for future work.

Summary updated

Abstract

Dense retrieval ranks one query vector against one document vector. On long documents, this interface can fail when a short but decisive span is weakened during document encoding before ranking. We study this failure mode as document-side early compression and introduce the Evidence Dilution Index (EDI) to measure how far a document-level representation falls below the strongest chunk-level evidence within the same gold document. Guided by this view, we propose DICE (Document Inference via Chunk Evidence), a training-free document-side strategy that splits documents into chunks, encodes them independently with a frozen model, and aggregates them back into a single vector while preserving the standard one-query-one-document interface. On LongEmbed, DICE improves retrieval across four backbones, with the largest gains on slices beyond 4k tokens: for Dream, Passkey >4k rises from 30.0 to 90.0 and Needle >4k from 23.3 to 74.0. Across 12,779 filtered samples, DICE yields lower EDI than the single-vector baseline in 92.8% of cases. These results establish document-level encoding as a practical and underexplored lever for long-document retrieval.
BibTeX
@misc{lyu2026lost,
  author = {Shanshan Lyu and Yiwei Wang and Yujun Cai and Jiafeng Guo and Shenghua Liu},
  title = {{Lost in a Single Vector: Improving Long-Document Retrieval with Chunk Evidence Aggregation}},
  year = {2026},
  eprint = {2606.18781},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2606.18781},
}

Loading related content...

Figure