Retrieval systems cut long documents into short chunks, and the cuts break meaning. A chunk that says "The city is also one of the states" no longer says which city; a clause that says "this requirement" has lost the section that defines it. We propose to repair each chunk before it is indexed by having a small sequence model rewrite it into a self-contained version. The rewrite is restricted: the model cannot write free text. It can only copy the chunk's own tokens, move a reading pointer backward or forward through the surrounding document, paste spans it finds there (for example the antecedent "Berlin" or the governing section header), and insert a handful of fixed connective words. Every token the model adds therefore traces to a specific location in the source document. As a side effect, the index also records which earlier passage each chunk depends on.
The design comes from operation sequence models in machine translation. These systems treat translation as a stream of discrete actions, either "emit this word pair" or "jump the source pointer by N", so that choosing words and reordering them happen in one left-to-right decision process rather than in separate pipeline stages. Fixing fragmentation has the same shape. To repair a chunk you must decide both what text to add and where in the document to fetch it from. We claim that a copy-and-jump action set recovers most of the retrieval benefit of free-form LLM rewriting, without inventing any content, at a fraction of the decoding cost. Unlike late chunking, it needs only a bounded window of preceding text, not the whole document in memory at once.
Core Claim
Contextual fragmentation (the loss of meaning when chunk boundaries separate content from its antecedents, headers, or dependencies) is currently handled in one of three ways. The first is embedding-time contextualization: Late Chunking runs a long-context encoder over the whole document before pooling chunk vectors, and Landmark Embedding reads sentence vectors off special boundary tokens inside a full context window. The second is query-time relinking: EBCAR reranks the top-k candidates with document-ID and position encodings and cross-passage attention, and KVLink stitches precomputed per-chunk caches using shifted positions and a few trainable link tokens. The third is heavy offline structuring: EnSI-RAG builds entity records, and GraphMPA builds summarized community graphs.
None of these produces a chunk whose text is self-contained and whose additions are guaranteed to be extractive. The embedding methods change vectors, not text, so the downstream generator still reads the ambiguous chunk. The graph methods require costly LLM parsing that is tailored to particular layouts. Our claim is narrower and testable. A copy/jump/paste operation grammar, trained by distillation from a free-form rewriter, matches late chunking and approaches free rewriting on cross-chunk retrieval benchmarks. It does so with zero unsupported insertions. Following Align, Write, Re-order, it should use relative pointer moves rather than absolute positions, and the advantage of relative moves should grow with document length.
The Operation Grammar
The model reads the current chunk plus a bounded window of preceding document text, for example the previous 4,000 tokens together with the header chain of the chunk's ancestors. It emits five kinds of actions:
- COPY(n) writes the next n tokens of the chunk to the output.
- JMP_BWD(k) and JMP_FWD(k) move a separate reading pointer by k sentence units, relative to its current position.
- PASTE(n) writes n tokens starting at the pointer.
- GEN(w) writes one word from a closed list of roughly fifty connectives and brackets, such as "(", ")", "of", "the" and ",".
These map onto the original operation sequence vocabulary of Can Markov Models Over Minimal Translation Units Help Phrase-Based SMT?: COPY plays the role of Generate, the jumps play the role of Jump(N), and GEN plays the role of Generate-Target-Only. The difference is that the source here is the document, the target is the repaired chunk, and no translation happens. Because COPY and PASTE cover whole runs of tokens, a typical repair should take tens of actions rather than hundreds of generated tokens. This addresses the decode-length inflation that the operation sequence literature flags as a cost.
After repair, each PASTE leaves an edge from the chunk to its source sentence. We store these edges as a lightweight dependency graph. It is produced as a byproduct of rewriting, with no separate graph-construction pass of the kind EnSI-RAG or GraphMPA require.
Research Program
Training data. Operation sequences have historically depended on external word alignments, and errors in those alignments corrupt the action targets. We face the same problem and handle it explicitly.
- A teacher LLM rewrites sampled chunks from the training corpora used in ConTEB (MLDR, SQuAD, NarrativeQA) into self-contained form, given the same bounded window.
- Each span the teacher inserts is matched to the window by exact match first and fuzzy match second.
- Insertions that match nowhere are deleted from the target. This is where teacher hallucinations are filtered out.
- Matched spans are converted to a minimal sequence of jumps and pastes.
Teacher-faithfulness statistic. The fraction of teacher insertions that cannot be grounded is itself a finding. It measures how often free rewriting adds content not found in the source.
Model. The student is a small encoder-decoder or decoder-only model in the 1B range. It is trained with cross-entropy on actions and pointer offsets.
Optional retrieval-reward stage. A second stage fine-tunes the student to raise the similarity between the repaired chunk and queries known to target it. This is kept separate so we can tell how much comes from distillation and how much from the retrieval signal.
Analysis. We study three things:
- what the model chooses to paste: antecedent noun phrases, header chains, or definitions;
- how far it jumps;
- whether long jumps concentrate in particular domains, such as Insurance or ESG reports, where definitions sit far from the clauses that use them.
Baselines
Naive chunking. Fixed-size, non-overlapping chunks embedded by the same off-the-shelf short-context embedder. This is the "do nothing" reference. We also run overlapping chunks at the same index size, because overlap is the cheapest repair available.
Late chunking. This is the strongest embedding-time competitor (Late Chunking). It addresses exactly the pronoun-antecedent failure we target. It reports a roughly 3.6% relative nDCG@10 gain over naive chunking on BeIR and raises the "Berlin" / "The city…" similarity from 0.75 to 0.85. We run it with its long-context encoder on identical chunk boundaries. We also run a window-truncated variant so that it has the same visible context as our model. We beat it if repaired chunks embedded by a short-context model match or exceed its nDCG@10 on ConTEB's out-of-domain sets, with lower peak indexing memory.
Free-form rewriting by the student architecture. This is the most direct threat. The same 1B model is trained on the unfiltered teacher rewrites with a normal text vocabulary. Base model, data, steps and tuning budget all match ours. We count our method as succeeding if it stays within about one nDCG@10 point of this baseline while having zero ungrounded insertions, where free rewriting shows a measurable rate of ungrounded insertions. The teacher LLM itself is reported as an upper reference.
Query-time reranking. EBCAR reports 64.92 average nDCG@10 across ConTEB at 29.33 queries per second on an A100. It is a reranker rather than a first-stage index, so we treat it as complementary. We run it on top of both naive and repaired first-stage retrieval with the same k = 20. A gain from repaired chunks under EBCAR would show that text-level repair supplies something embedding-space reranking cannot recover. EBCAR's own paper names its embedding bottleneck as a limitation.
Ablations. Four ablations isolate the key ingredients:
- absolute pointer indices in place of relative jumps;
- removing GEN, so the model can only copy and paste;
- removing the window and keeping only the header chain;
- dropping the dependency edges at retrieval time.
Why not just…?
Why not just use late chunking? It changes only the vector. The generator downstream still receives "The city is also one of the states" without "Berlin," so answer attribution can still fail even when retrieval succeeds. It also needs the full document in one encoder pass, which causes the memory spikes its own paper notes and does not fit streaming ingestion. Still, on pure retrieval metrics it may tie us. The Loong downstream experiment, which feeds repaired versus original chunk text to the same generator, is what separates the two.
Why not just let an LLM rewrite chunks freely? This is the real threat, and we treat it as a co-equal arm rather than a strawman. Free rewriting can paraphrase and merge information in ways our grammar cannot, so it may win on retrieval. Our case rests on three measurable properties: ungrounded-insertion rate, decode cost, and provenance edges. The teacher-faithfulness statistic quantifies the first. If free rewriting shows near-zero ungrounded insertions and matches our cost, the operation grammar has no reason to exist, and we would report that.
Why not just rerank with EBCAR or stitch caches with KVLink at query time? Both pay their cost at every query. KVLink also assumes chunks are joined monotonically and reports partial recovery of the accuracy lost from naive cache reuse (KVLink). Repair is a one-time indexing cost and composes with either method. The EBCAR-on-top experiment shows whether the gains stack or overlap.
Why not just feed the whole document to a long-context model? When Does Divide and Conquer Work for Long Context LLM? reports that full-attention performance degrades sharply at long lengths, and that chunked processing with smaller models can win. Repair makes the chunked side of that tradeoff less isolated, so it targets the silo failure that paper identifies.
Proposed Experiments
Primary benchmark. ConTEB, using average nDCG@10 over its out-of-domain sets (COVID-QA, ESG Reports, Football, Geography, Insurance) plus MS-MARCO v2 as a check on general retrieval. BeIR subsets serve as a no-harm check, since most of their chunks need no repair.
Downstream benchmark. Loong, using a fixed generator and the same retrieval budget across arms. The published reference points are EnSI-RAG's 78.24 average accuracy across Loong and Oolong, and Disco-RAG's reported 12.74-point LLM-score gain. We do not expect to match systems built on offline parsing. We do expect repaired chunks to raise a plain RAG pipeline on Loong's spotlight-locating and comparison tasks.
Additional metrics:
- ungrounded-insertion rate, judged by string match and checked by entailment;
- actions and generated tokens per chunk;
- peak indexing memory;
- jump-target correctness, judged by annotators on 300 sampled repairs: does the pasted span actually resolve the reference?
Decisive checks. There are four:
- Repaired chunks with a short-context embedder match or beat late chunking on ConTEB out-of-domain sets.
- The operation model lands within about one point of free rewriting, with zero ungrounded insertions versus a nonzero rate for free rewriting.
- Relative jumps beat absolute indices, with the gap widening as documents grow longer. This replicates the generalization finding of Align, Write, Re-order in a new setting.
- Under a streaming setup with a fixed window, our method degrades more gracefully than window-truncated late chunking.
If the first two checks fail, the operation grammar is the wrong abstraction for this problem, and the paper becomes a measurement study of how faithful free rewriting is.
References
- Late Chunking: Contextual Chunk Embeddings Using Long-Context Embedding Models
- Landmark Embedding: A Chunking-Free Embedding Method For Retrieval Augmented Long-Context Large Language Models
- Embedding-Based Context-Aware Reranker
- KVLink: Accelerating Large Language Models via Efficient KV Cache Reuse
- EnSI-RAG: Entity-Structure-Indexed Retrieval-Augmented Generation for Long-Document Question Answering
- A Comprehensive Graph Framework for Question Answering with Mode-Seeking Preference Alignment
- Align, Write, Re-Order: Explainable End-to-End Speech Translation via Operation Sequence Generation
- Can Markov Models Over Minimal Translation Units Help Phrase-Based SMT?
- When Does Divide and Conquer Work for Long Context LLM? A Noise Decomposition Framework
- Disco-RAG: Discourse-Aware Retrieval-Augmented Generation