Uncertainty-Aware Hybrid Retrieval for Long-Document RAG

Hoin Jung, Xiaoqian WangarXiv 2026 · arXiv

Here is a concise, engaging blog-style summary of the paper.

TL;DR

This paper introduces UMG-RAG and its extension UMGP-RAG, training-free frameworks designed to solve the "granularity tradeoff" in long-document Retrieval-Augmented Generation (RAG). By dynamically measuring the uncertainty of dense and sparse retrievers across multiple chunk sizes, the system fuses the most reliable evidence and promotes precise, fine-grained hits into coherent parent chunks. This plug-and-play approach significantly boosts LLM generation quality while slashing downstream generation time and memory overhead.

What Problem Does This Paper Address?

Retrieval-Augmented Generation (RAG) systems generally rely on breaking long documents down into text chunks. However, system designers face a frustrating goldilocks dilemma regarding chunk size:

  • Coarse-grained chunks (large text blocks): These preserve context well, but they introduce irrelevant noise. This noise dilutes the actual answer-bearing evidence and triggers the notorious "lost-in-the-middle" phenomenon, where LLMs struggle to utilize information buried in long prompts.
  • Fine-grained chunks (sentences or short phrases): These are highly compact, but they are difficult for retrievers to find because short snippets often lack the rich semantic or lexical cues needed to match a user's query.

Existing solutions either try to fix the LLM's attention span post-retrieval or build complex, heavy indexing structures like knowledge graphs. This paper addresses the retrieval-stage decision itself: How can a system dynamically decide whether to retrieve a compact local chunk, a broader context, or a mix of both, without requiring expensive model retraining?

Methods

The authors propose Uncertainty-aware Multi-Granularity RAG (UMG-RAG), a training-free, plug-and-play framework that treats chunk granularity as a query-specific reliability estimation problem.

  1. Multi-Granularity Indexing: Documents are segmented into overlapping sentence-level chunks across multiple granularities (specifically 2, 4, 8, 16, and 32 sentences).
  2. Hybrid Experts: The system employs both a dense retriever (for semantic matching) and a sparse retriever (for lexical/entity matching) to score query-chunk pairs across all granularities.
  3. Uncertainty-Aware Fusion: Instead of using a fixed weight to combine dense and sparse scores, UMG-RAG evaluates the sharpness of each expert-granularity score distribution. A peaked distribution indicates low entropy (high confidence), while a flat distribution indicates high entropy (high uncertainty). The system calculates normalized entropy to dynamically weight and fuse candidates into a single ranked list.
  4. Parent Promotion (UMGP-RAG): To ensure the LLM receives coherent text, this variant uses ultra-fine chunks (2 or 4 sentences) purely as precise "locators." Once a small chunk is highly ranked, it is automatically promoted to its broader "parent" chunk (8 sentences). An overlap-aware deduplication step then filters out redundant text spans.
[Query] ➔ Evaluated by Dense & Sparse Experts across granularities (2, 4, 8, 16, 32 sentences)
           ➔ Confidence calculated via Score Distribution Entropy
           ➔ Candidates Fused ➔ Fine Chunks Promoted to Parent Chunks ➔ Deduplicated ➔ [LLM]

Datasets

The framework was evaluated on two widely recognized question-answering benchmarks:

  • Natural Questions (NQ): Focuses on factual question answering over external knowledge.
  • HotPotQA: Requires multi-hop reasoning and more complex, compositional evidence retrieval.

For both datasets, the authors constructed task-specific retrieval corpora using the KILT framework validation split, mixed with 10,000 randomly sampled distractor Wikipedia documents to simulate a realistic, noisy retrieval environment.

Key Insights and Ideas

  • Distribution Shape as a Trust Signal: The core "aha moment" is that a retriever's confidence can be read directly from the entropy of its top outputs. If a retriever gives a single chunk a massive score and the rest low scores, it knows exactly what it's looking for. If it gives 100 chunks similar scores, it's guessing.
  • Retrieval vs. Generation Coherence: The best chunk size for finding information isn't always the best size for reading information. Separating the "locator" signal (fine-grained retrieval) from the "reader" context (parent promotion) yields the best of both worlds.
  • Shifting the Computational Burden: By spending slightly more compute during the retrieval preprocessing stage to evaluate multiple granularities, the system constructs a highly compact prompt. This drastically reduces the time and memory the LLM spends during the generation phase.

Key Results

  • Superior Generation Quality: Across multiple retrievers (BERT, BGE-M3, Qwen3-Embedding) and generators (Qwen2.5-3B, Llama-3.2-3B), UMGP-RAG consistently achieved the highest or highly competitive generation performance (measured by F1 score and Answer Recall). For instance, on the NQ dataset using Qwen3-Embedding and Llama-3.2-3B-Instruct, UMGP-RAG achieved a generation F1 of 0.5220, outperforming standard RAG (0.4939) and standard Hybrid RRF fusion (0.4916).
  • Downstream Efficiency Gains: According to the computational cost analysis, UMGP-RAG achieved the lowest generation times (0.33 seconds/question) and the lowest peak memory footprints (under 7000 MiB) compared to all long-context baselines, proving that compact contexts save heavy LLM compute.

Limitations and Future Work

  • Retrieval Preprocessing Latency: The primary drawback is the upfront cost. Querying both dense and sparse retrievers across five different granularities increased retrieval preprocessing time to roughly 5.36 seconds per question in testing.
  • Garbage In, Garbage Out: Because the framework is entirely training-free, it is fully dependent on its underlying base retrievers. If both the dense and sparse retrievers fail to surface the correct evidence in their initial top-100 candidate pools, the uncertainty-aware fusion cannot recover the missing information.

Future work could explore optimizing the multi-granularity retrieval step to lower initial latency while maintaining high precision.

Summary updated

Abstract

Retrieval augmented generation (RAG) depends critically on the quality and granularity of retrieved evidence. Large retrieval units preserve context but often introduce irrelevant content, which can dilute answer bearing evidence and worsen long context utilization. Fine-grained units are more compact, but they may be difficult to retrieve reliably because short chunks can lack semantic, lexical, or bridging cues needed to match the query. We propose Uncertainty-aware Multi-Granularity RAG (UMG-RAG), a training-free hybrid retrieval framework that treats chunk granularity as query-specific reliability estimation. Instead of training a new retriever or modifying the generator, UMG-RAG uses existing dense and sparse retrievers as complementary experts across multiple chunk granularities. For each query, it converts each expert-granularity score list into an evidence distribution, estimates reliability from distribution entropy, and fuses candidates according to query-specific semantic, lexical, and granularity confidence. We further introduce UMGP-RAG, a parent promotion variant that uses fine-grained hits to locate relevant evidence while returning broader non-redundant parent chunks for local coherence. Experiments on question answering benchmarks show that uncertainty-aware fusion and parent promotion improve generation quality while maintaining a lightweight, plug-and-play retrieval pipeline.
BibTeX
@misc{jung2026uncertainty,
  author = {Hoin Jung and Xiaoqian Wang},
  title = {{Uncertainty-Aware Hybrid Retrieval for Long-Document RAG}},
  year = {2026},
  eprint = {2606.13550},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2606.13550},
}

Loading related content...

Figure