StitchRAG: Constructing Compact and Evidence-Preserving Contexts for RAG with Diffusion-Based Stitching

Minsang Kim, Junpyo Lim, Heeje Cho, Seungjun BaekarXiv 2026 · arXiv

TL;DR

StitchRAG is a novel framework that optimizes Retrieval-Augmented Generation (RAG) by dynamically compressing retrieved passages and "stitching" the remaining evidence together using text diffusion. By treating context construction as a utility maximization problem, it creates compact, information-dense inputs that improve accuracy and robustness to noise while significantly reducing computational overhead compared to iterative multi-step RAG methods.

What Problem Does This Paper Address?

While Retrieval-Augmented Generation (RAG) has become the standard for grounding Large Language Models (LLMs) in factual data, it faces a "context rot" problem. Simply retrieving more documents often leads to long, noisy contexts that confuse the model—a phenomenon often called the "lost in the middle" effect.

Existing solutions generally fall into two camps: multi-step reasoning (which is slow and expensive) and context compression (which often uses fixed ratios and can break the semantic flow of the text). The authors of StitchRAG identify a critical gap: compression should be context-dependent, and the resulting fragmented evidence needs to be reconstructed into a coherent "reasoning chain" for the LLM to process it effectively.

Methods

StitchRAG operates in three main stages, moving from raw retrieved passages to a "stitched" compact context:

  1. Optimal Context Retention: Instead of a one-size-fits-all compression ratio, StitchRAG calculates a dynamic budget ($r^*$) for each passage. It formulates this as a utility maximization problem: balancing the relevance utility (how much info the passage provides for the query) against the computation cost (which scales quadratically with length due to self-attention).
  2. Hybrid Anchor Extraction: To ensure critical evidence isn't pruned, the system identifies "anchors"—key terms or spans. It uses a hybrid strategy:
  3. Reasoning-centric (AR-based): The LLM generates potential answer candidates.
  4. Query-centric (TF-IDF): Lexical matching to capture rare entities. Sentences containing these anchors are protected and prioritized during selection.

  5. Diffusion-Based Stitching: This is the framework's signature move. After pruning irrelevant sentences, the remaining fragments are concatenated. To restore coherence, StitchRAG inserts "masked" tokens between fragments and uses a discrete text diffusion model (like LLaDA or Dream) to fill them in.

Figure 2: Overview of StitchRAG. Given a query and a set of retrieved passages, StitchRAG 1) computes a per-passage dynamic retention ratio r∗, 2) extracts passage sentences containing key terms, called anchors, and aggregates them under the token budget constrained by r∗, and 3) generates bridging text between the aggregated sentences using a diffusion LLM. Finally, the constructed context is fed into the LLM to generate the answer.

Datasets

The authors evaluated StitchRAG on five diverse open-domain QA benchmarks:

  • HotpotQA & 2WikiMultiHopQA: Focused on multi-hop reasoning across multiple documents.
  • Natural Questions (NQ) & TriviaQA: Standard factoid-based open-domain QA.
  • PopQA: An entity-centric benchmark designed to test long-tail factual knowledge.

Key Insights and Ideas

  • Diffusion as Interpolation: Most LLMs are autoregressive (left-to-right). However, "stitching" is an interpolation problem—you need to know what comes before and after the gap. Diffusion models, with their bidirectional attention, are naturally suited for this "in-filling" task.
  • The "Verbatim Constraint": During the stitching process, the original evidence tokens are frozen. The diffusion model only generates the "connective tissue." This prevents the system from hallucinating new facts while improving the readability of the context.
  • Robustness to Retrieval Depth: A major "aha" moment is shown in Figure 3. As you increase the number of retrieved passages ($K$), standard RAG performance drops because of the added noise. StitchRAG's performance remains nearly flat, effectively decoupling generation quality from the raw volume of retrieved data.
Figure 3: Robustness to increased retrieval passages. Average F1 across four QA benchmarks (Hotpot, NQ, TriviaQA, 2Wiki) as the retrieval number of Top-K in- creases (Top-10/20/30). While standard RAG degrades sharply at larger K due to longer and noisier contexts, StitchRAG remains stable, mitigating retrieval noise and context-length effects.

Key Results

StitchRAG consistently outperformed both standard RAG and state-of-the-art compression/multi-step baselines (like LongLLMLingua, Self-RAG, and DRAG).

  • Accuracy: On HotpotQA, it improved over standard RAG by +2.4 EM and +2.3 F1. It ranked first on 6 out of 10 evaluation metrics across all datasets.
  • Efficiency: While slower than a simple one-pass RAG, StitchRAG is significantly faster than iterative methods. For example, it achieved a latency of ~10.6s, compared to ~24.6s for IRCoT and ~27.4s for DRAG.
  • Compression: On average, it reduced context length to ~72% of the original size while actually improving the model's ability to find the correct answer.
Table 1: Comparison of retrieval-augmented generation methods. (✓: Retrieval used, ✗: No retrieval). Bold indicates the best performance. and Underline denotes the second-best performance.

Limitations and Future Work

  • Model Overhead: The framework requires multiple model calls (relevance probing, anchor generation, and diffusion denoising). While the authors optimized this with vectorized implementations and batching, it still carries more overhead than "naive" RAG.
  • Diffusion Model Maturity: The approach is currently limited by the capabilities of existing discrete diffusion LLMs. As these models mature, the authors envision more aggressive "context refinement"—not just stitching fragments, but elegantly rewriting and reorganizing entire contexts.
  • Fixed Hyperparameters: The cost-penalty hyperparameter ($\beta$) was set to 0.25 across experiments; exploring how to tune this dynamically based on hardware constraints or latency requirements is a potential next step.

Summary updated

Abstract

Retrieval-augmented generation (RAG) improves factuality by grounding large language models (LLMs) in external evidence. However, long retrieved contexts incur computational overhead and can even hurt performance due to noisy or irrelevant content. We propose \textsc{StitchRAG}, a framework for constructing compact context in RAG. \textsc{StitchRAG} builds a faithful, information-dense context while dynamically compressing the retrieved passages. We adapt the size reduction ratio to each passage by formulating a utility maximization trading-off the retained information and computation costs. We then extract answer candidates and query-related keywords from passages as anchors and collect passage sentences containing anchors under the derived size constraint. Finally, we connect or stitch the collected sentences together using text diffusion to form a coherent and natural context. Experiments on five open-domain QA benchmarks show that \textsc{StitchRAG} consistently outperforms diverse multi-step and compression baselines. Extensive ablations show the effectiveness of dynamic size allocation, anchor preservation, and diffusion-based stitching.
BibTeX
@misc{kim2026stitchrag,
  author = {Minsang Kim and Junpyo Lim and Heeje Cho and Seungjun Baek},
  title = {{StitchRAG: Constructing Compact and Evidence-Preserving Contexts for RAG with Diffusion-Based Stitching}},
  year = {2026},
  eprint = {2611.22766},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.22766},
}
Figure