TL;DR
Standard dense retrieval models struggle with long documents because short, critical pieces of evidence get diluted when compressed into a single vector. This paper introduces DICE (Document Inference via Chunk Evidence), a training-free method that chunks long documents, encodes them independently using local position IDs, and aggregates them back into a single vector. DICE dramatically boosts long-document retrieval performance across multiple model architectures while preserving the standard, highly scalable one-query-one-document interface.
What Problem Does This Paper Address?
Dense retrieval systems typically encode an entire document into a single vector to rank it against a query vector. While modern embedding models boast massive context windows (e.g., 4k to 32k+ tokens), their retrieval performance still degrades sharply on long texts.
The authors identify the root cause as document-side early compression (or evidence dilution). When a query's answer lies within a short, localized span buried inside thousands of irrelevant tokens, encoding the entire document at once mixes the crucial signal with background noise before any query comparison happens. As a result, the gold document receives a low similarity score.
To quantify this phenomenon, the authors introduce the Evidence Dilution Index (EDI), a diagnostic metric that measures how far a document-level vector representation falls below its strongest chunk-level evidence.
Methods
To combat evidence dilution without sacrificing the simplicity of the single-vector interface, the authors propose DICE (Document Inference via Chunk Evidence). DICE is a training-free, document-side encoding strategy consisting of three steps:
- Chunking: The long document is split into non-overlapping chunks of a fixed token size $k$ (defaulting to 1024 tokens).
- Local Position Encoding: Each chunk is encoded independently using a frozen embedding model. Crucially, the positional indices are reset to start from zero for each chunk, treating them as self-contained context windows.
- Aggregation: The resulting chunk embeddings are fused into a single document vector using a query-independent aggregation function $g(\cdot)$. Through empirical testing, the authors found that simple mean pooling works best, preserving evidence from all chunks equally.
Because all modifications happen purely on the document-encoding side, the query path and the standard vector-search infrastructure remain completely untouched.
Datasets
The authors evaluated DICE using two primary benchmarks:
- LongEmbed: A benchmark specifically designed for long-context retrieval, featuring two synthetic tasks (Passkey and Needle in a haystack) and four real-world retrieval datasets (NarrativeQA, QMSum, SummScreenFD, and WikiMQA).
- FollowIR: A benchmark used for secondary transfer evaluation to determine if the gains generalize to instruction-following retrieval tasks.
The evaluations spanned four diverse model backbones: Dream (diffusion-based), Llama3-8B, Mistral-7B, and Qwen2.5-7B (autoregressive models converted to bidirectional embedders).
Key Insights and Ideas
- Capacity vs. Representation: The long-document retrieval bottleneck is not a context-window capacity issue, but a representational one. Models have the space for long contexts, but global compression washes out localized details.
- Simplicity Wins: More sophisticated aggregation techniques, such as max-pooling or selecting chunks based on embedding norms (top-$k$), actually degrade performance on real-world tasks. Conservative mean pooling is the most robust strategy because it preserves distributed evidence without over-indexing on noisy peaks.
- Local Positions are Crucial: Resetting position encodings for each chunk outperforms keeping absolute document offsets. Local resets ensure chunks are processed with high fidelity, whereas absolute offsets can push tokens into ranges where the model's positional embeddings are less robust or out-of-bounds.
Key Results
DICE delivers massive performance leaps, particularly on the longest and most challenging document slices:
- Synthetic Tasks (>4k tokens): For the Dream backbone, Passkey >4k accuracy soared from 30.0% to 90.0%, and Needle >4k accuracy jumped from 23.3% to 74.0%.
- Mitigating Length Degradation: As context length scales up to 32k tokens, standard single-vector encoding performance collapses toward zero, while DICE maintains stable, high retrieval accuracy.
- EDI Validation: Across 12,779 evaluated samples, DICE successfully achieved a lower Evidence Dilution Index than the single-vector baseline in 92.8% of cases, proving mathematically that it keeps the final document vector closer to the actual evidence.
Limitations and Future Work
While DICE provides a powerful, training-free fix, it comes with a few trade-offs:
- Computational Cost: Encoding a document chunk-by-chunk increases document-side compute time by roughly 3× to 4×. This trade-off is highly acceptable for offline indexing (where documents are encoded once and searched many times) but less ideal for real-time document streaming.
- Query-Blind Aggregation: Because aggregation happens before the query is seen, DICE cannot dynamically weight chunks that are highly relevant to a specific user prompt.
The authors note that exploring learned, query-independent compression techniques to pack multi-vector chunk representations into a single vector remains an exciting direction for future work.
Summary updated