Optimizing Retrieval Granularity for Contextual Language Models

181 concepts136 papers1 proposal

When building Retrieval-Augmented Generation (RAG) systems or processing long documents, we inevitably face a frustrating architectural choice: How large should our text units be?

If we partition a document into small segments (such as individual sentences or short token windows), our retrieval system becomes highly precise. It can pinpoint exact facts without drowning the downstream Large Language Model (LLM) in irrelevant text. However, these tiny segments are stripped of their surrounding context, often rendering them semantically ambiguous or incomplete.

Conversely, if we retain large segments (such as full paragraphs or entire documents), we preserve the narrative structure and global context. But this introduces substantial noise, increases computational overhead, and risks model confusion or hallucinations due to the dilution of specific target information.

To resolve this tension, a prominent research direction focuses on retrieval granularity optimization. Rather than relying on arbitrary, fixed-size text chunks, modern architectures dynamically decouple, contextualize, and adapt the size of the retrieval unit to balance precision with contextual integrity.


The Core Mechanics of the Granularity Trade-Off

To understand why fixed chunking fails, consider how text embeddings work. When a text segment is embedded in isolation, its vector representation must capture its entire meaning. As noted by Michael Günther et al. in Late Chunking: Contextual Chunk Embeddings Using Long-Context Embedding Models, the smaller the retrieval unit, the more vulnerable it is to losing critical semantic meaning when separated from its global document context.

Traditional fixed-length chunking creates a rigid boundary before the model ever sees the text. This introduces two primary failure modes:

  1. Semantic Dilution: Large passages contain irrelevant information that distracts the model's attention and introduces noise.
  2. Contextual Fragmentation: Small segments lack the necessary background (such as pronoun referents or temporal framing) to be fully understood by either the retriever or the generator.

Shifting from Arbitrary Tokens to Semantic Atoms

One powerful solution is to replace arbitrary token counts with self-contained semantic units. Instead of cutting text mid-sentence, systems can break documents down into propositions—defined as atomic, self-contained factoids.

In Dense X Retrieval: What Retrieval Granularity Should We Use?, Hongming Zhang et al. demonstrate that indexing text at the proposition level significantly boosts retrieval accuracy, particularly for rare or "long-tail" knowledge. Because each proposition is fully self-contained yet minimal, it avoids the noise of a full paragraph while bypassing the fragmentation of a raw sentence.

Building upon this atomized approach, frameworks like the one proposed by Jinyu Guo et al. in HASH-RAG: Bridging Deep Hashing with Retriever for Efficient, Fine Retrieval and Augmented Generation pair proposition-level chunking with deep binary hashing. This combination drastically reduces storage and retrieval latency while maintaining high factual precision.


Late Chunking: Global Context for Local Units

If small chunks are ideal for precision but terrible for context, can we embed text globally but retrieve it locally? This is the intuition behind Late Chunking.

Developed by Michael Günther et al., Late Chunking leverages long-context embedding models to condition the representations of small text units on the entire document. Instead of chunking the text before passing it to an embedding model, the entire document is processed first. This allows the model's attention mechanisms to contextualize every token across the global document scope. Only after this contextualization are the token embeddings pooled into smaller, granular chunk representations.

Traditional Chunking:
Raw Text ──> Split into Chunks ──> Embed Chunks Individually ──> Isolated Vectors

Late Chunking:
Raw Text ──> Long-Context Model (Global Attention) ──> Pool into Chunks ──> Contextualized Vectors

As a result, a sentence buried on page 10 retains its relationship to the introduction on page 1, allowing high-precision retrieval without semantic isolation.


Dynamic and Hierarchical Adaptation

Not every query requires the same level of detail. A question about a specific statistic demands granular precision, while a query summarizing a trend requires a broader view. Consequently, researchers have designed systems that adapt their granularity dynamically during inference.

Hierarchical Merging

One approach is to store text at multiple levels of granularity simultaneously. In HiChunk: Evaluating and Enhancing Retrieval-Augmented Generation with Hierarchical Chunking, Xing Sun et al. introduce an "Auto-Merge" algorithm. The system retrieves highly granular chunks to ensure matching accuracy but automatically merges adjacent chunks into a larger, cohesive paragraph if multiple fragments from the same section are deemed relevant.

Multi-Granular Graphs

For complex, multi-hop reasoning, frameworks like MEGRAG: Multi-Granular Evidence Graphs for Answer-Aware Multi-Hop RAG construct graphs containing multiple granularities—ranging from compact semantic triples to full text passages. The system dynamically navigates this graph, selecting the precise level of detail needed at each individual step of the reasoning chain.

Dynamic Routing

Alternatively, the system can determine the optimal chunk size before retrieval even begins. Hanwen Liu et al. developed Mix-of-Granularity: Optimize the Chunking Granularity for Retrieval-Augmented Generation, which uses a lightweight router to analyze incoming queries and predict the ideal chunk size required to fulfill the request, optimizing the balance between noise reduction and context preservation on the fly.


Aligning Granularity with Downstream Tasks

Ultimately, the optimal retrieval unit depends heavily on the cognitive operation the language model is expected to perform.

Zhiyu Li systematically mapped this relationship in CRUD-RAG: A Comprehensive Chinese Benchmark for Retrieval-Augmented Generation of Large Language Models. The benchmark reveals a clear division: large document chunks are superior for creative generation and synthesis tasks because they preserve narrative flow and structural continuity. Conversely, small, highly focused chunks are significantly more effective for question-answering and fact-checking, where isolating specific facts minimizes the risk of model distraction.

This operational alignment is also critical when managing long-context models. While it is tempting to feed massive documents directly into an LLM, Zhen Xu demonstrates in When Does Divide and Conquer Work for Long Context LLM? A Noise Decomposition Framework that extreme input lengths cause a performance collapse ("brain fog") due to accumulated noise. By establishing a principled balance between processing small, manageable segments and maintaining enough context for global reasoning, systems can bypass the limitations of both over-fragmentation and under-partitioning.

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 123 papers
  2. Version 1 · 2026-10-04 · 172 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_retrieval_granularity_optimization_3f5a9efb,
  title = {retrieval granularity optimization},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/retrieval-granularity-optimization/rdr_3f5a9efbc8e949db941b883c11588253},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 2, frozen text):

@misc{lacuna_retrieval_granularity_optimization_3f5a9efb_v2,
  title = {retrieval granularity optimization},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/retrieval-granularity-optimization/rdr_3f5a9efbc8e949db941b883c11588253/v/2},
  note = {Version 2, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake