Learning Where to Cut: Retrieval-Trained Semi-Markov Chunking

Retrieval systems for long documents cut each document into short passages before embedding them. Those cuts are almost always made blindly, at a fixed token count or at a similarity threshold. When a cut lands in the middle of a definition, a legal condition, or a pronoun's link to its referent, the fact it breaks becomes unretrievable. Context-aware embedding and reranking methods repair some of this damage after the fact, but every one of them accepts the boundaries as given.

We propose to learn the boundaries themselves. The tool is a semi-Markov conditional random field (semi-CRF), a sequence model that labels whole variable-length segments rather than single tokens. A semi-CRF can score every candidate chunk as a unit, include a prior over chunk lengths, and find the best segmentation of a whole document by dynamic programming. We train it with a retrieval signal: a chunk is good if it keeps a question's supporting evidence in one piece and its embedding can be found by that question.

Core Claim

Contextual fragmentation is partly a boundary-placement problem. Boundary placement can be posed as segment-level structured prediction, with weak supervision taken from question-answering data that marks where the evidence lies. We claim that a semi-CRF over sentences has three ingredients that together reduce the rate at which evidence is split across chunks, and improve retrieval at a fixed budget of retrieved tokens, compared with fixed windows, threshold-based semantic chunking, and the same dynamic program with untrained scores. The three ingredients are:

  • span scores computed from document-contextualized token embeddings;
  • an explicit chunk-length prior;
  • training against an objective that forbids cutting through evidence.

The pieces exist separately, and their roles differ:

  • Supplies the dynamic programming and the joint segment scoring. Semi-Markov models for segmentation (Smola) give exact decoding over variable-length segments.
  • Supplies the length prior. Duration modeling (DM-SMCRFs) showed that keyphrase lengths follow domain-specific Gamma or Gaussian distributions. Adding that distribution as a potential improved F1 by about 10% over token CRFs. That work stays within sentences; we move the duration idea to passage-scale units.
  • Supplies the span representation. Late Chunking encodes the full document before mean-pooling token states into chunks, so that "the city" carries "Berlin". It takes chunk boundaries as fixed inputs; we make them the output.
  • Motivates the problem. PIC groups sentences whose similarity to an LLM-written summary exceeds the document mean. Its gains are real, but it is greedy, it has no length control, and by the authors' own ablation it is sensitive to the threshold.
  • Shows that boundary-blind repair has limits. EBCAR mixes information across pre-cut chunks from the same document in embedding space. It cannot undo a bad cut.

The closest structural precedent is the section-title system of Gehrmann et al.. It uses a semi-CRF, but only to compress single sentences, with paragraph boundaries fixed.

Research Program

Segmentation unit and model. The unit of segmentation is the sentence, not the token. This choice also settles the scaling question. With $S$ sentences and a maximum chunk length of $D$ sentences (set by the embedder's input limit, for example 512 tokens), exact decoding costs $O(S \cdot D)$. A few-hundred-sentence document is therefore cheap without any pruning.

We deliberately do not use the null-span filtering of Filtered Semi-CRF. A salience filter would discard exactly the transitional sentences that DAPR identifies as the missing context behind 53.5% of retrieval errors (DAPR). Every sentence must belong to some chunk; the model only decides where the cuts go.

Scoring a segmentation. A segmentation $\mathbf{s}$ of document $x$ is a sequence of chunks $(u_p, v_p)$ covering all sentences. Its score is

$$\mathrm{score}(\mathbf{s}) = \sum_p \Big[ f_\theta(h_{u_p:v_p}) + \log P_\phi(v_p - u_p + 1) + g_\theta(h_{v_p}, h_{v_p+1}) \Big].$$

The terms are defined as follows:

  • $h_{u:v}$ is the mean of late-chunked token states over sentences $u$ through $v$.
  • $f_\theta$ is a small network that scores how self-contained a chunk is.
  • $P_\phi$ is a learned discrete length distribution, initialized from a Gamma fit, following DM-SMCRFs.
  • $g_\theta$ scores the boundary between the last sentence of one chunk and the first of the next.

Long documents are contextualized in overlapping macro-chunks, as in Long Late Chunking. This also limits the long-input degradation that the divide-and-conquer noise analysis attributes to excessively long single-shot inputs.

Training signal. No gold chunk boundaries exist, so we train from QA data with located evidence. ConTEB's training splits (MLDR, SQuAD, NarrativeQA) and NQ provide question–evidence pairs. Training combines two losses.

  1. Constrained marginal-likelihood loss. This is $\log Z(x) - \log Z_{\mathcal{C}}(x)$, where $Z$ sums the exponentiated score over all segmentations and $Z_{\mathcal{C}}$ sums only over segmentations that cut through none of the document's evidence regions. Both are computed by the same forward algorithm.
  2. Expected retrieval loss. The semi-CRF's chunk marginals weight a contrastive loss. Its positive is the chunk containing the evidence, and its negatives are other chunks in the batch. This pushes toward chunks that are small enough to be precise, not merely large enough to avoid cuts.

Indexing and deployment. At indexing time we Viterbi-decode each document, pool each chunk from its late-chunked states, and store ordinary vectors. The output is a standard approximate-nearest-neighbor index. Unlike CFIC, which decodes evidence spans from a single document held in an LLM's context, this scales to a corpus. Any downstream reranker, including EBCAR, runs unchanged on top.

Analysis. Beyond accuracy, we will examine where the model places boundaries relative to headings and discourse markers. We will also check whether learned length distributions differ by domain, as DM-SMCRFs found for keyphrases.

Baselines

Do-nothing reference. Fixed 256-token windows embedded independently. We also report the same windows with 50% overlap, accepting the doubled index.

Isolated-passage dense retrieval. DRAGON+ on DAPR, which DAPR reports at 0.0 nDCG@10 on NQ-hard. This fixes the floor for context-dependent queries.

Strongest chunking baseline: Late Chunking with fixed sentence-group or 256-token boundaries (Late Chunking), which reports about a 3.6% relative nDCG@10 gain over naive chunking on BeIR. It is the strongest competitor because it shares our encoder exactly. Any gain we show over it is attributable to boundary placement alone.

Strongest learned-free dynamic segmentation baseline: PIC (PIC), which reports a 2-point Hits@5 gain on WebQ and a 2.0-point average EM gain with Qwen2.5-7B-Instruct. We will run it with the same sentence embedder and the same summary LLM setting the paper describes. We will report it both at its published threshold and at the best threshold found on our development split.

Strongest end-to-end system: EBCAR (EBCAR), at 64.92 average nDCG@10 on ConTEB. It is a reranker, so the fair comparison runs EBCAR at $k=20$ on top of fixed chunks versus on top of our chunks. The same first-stage embedder, same training data, and same tuning budget apply to both. Absolute late-chunking numbers on ConTEB: no reported result found on Lacuna for that pairing, so we will produce them.

Ablations that isolate the key ingredient. The most important is the same semi-CRF dynamic program with the same length prior but untrained scores. In it, $f$ is cosine cohesion between adjacent sentences, which amounts to optimal similarity-based segmentation with a length penalty. Further ablations remove:

  • the length prior;
  • the retrieval loss (constraint loss only);
  • the constraint loss (retrieval loss only);
  • late contextualization, using isolated sentence embeddings inside the chunk scorer.

What counts as beating a baseline. All comparisons are made at matched retrieved-token budget, since larger chunks trivially cover more evidence. Beating a baseline means higher evidence-token recall at 1k and 2k tokens of retrieved text, and higher nDCG@10 under overlap-mapped relevance, on the out-of-distribution ConTEB domains and on DAPR. The margin must exceed seed variance across three runs.

Why not just…?

Why not just prepend document titles to every chunk? DAPR found title prepending beats coreference resolution on NQ-hard, so it is a serious baseline. We include it as a condition crossed with every chunker. DAPR also reports that it hurts in Genomics, and it does nothing when an argument is split mid-thought. Prepending and learned boundaries are complementary. The experiment that settles their relationship is the titles × chunker grid on DAPR: if titles plus fixed windows matches titles plus learned chunks, our contribution is moot for that regime.

Why not just run optimal dynamic-programming segmentation over sentence similarity with a length penalty? This is the real threat. It is exactly our untrained ablation, which shares the decoder and the length prior and differs only in the learned, retrieval-trained scores. If it matches the full model, the paper's claim reduces to "a global length-aware decoder beats PIC's greedy threshold." That would still be a useful finding, and we would report it as such.

Why not just index sentences and expand to neighbors at query time? Retrieving small units and returning a fixed window around each hit avoids choosing boundaries. However, isolated sentences are exactly the units whose embeddings lose their referents, which is the failure DAPR measures. The expansion window is also fixed and blind to structure. We will run sentence-level late chunking with ±2 and ±4 sentence expansion at the same token budget. If it wins on evidence recall, learned chunking is unnecessary for first-stage retrieval.

Why not just use a long-context LLM over the whole document, or CFIC? For corpus-scale search this is not an option. The divide-and-conquer analysis also reports that GPT-4o-mini with chunking outperformed single-shot GPT-4o on a 128k-token task, so full-context reading is not a free upper bound even per document. We use CFIC only as a per-document reference on LongBench-style tasks, not as a retrieval competitor.

Proposed Experiments

Training and evaluation data. Training uses ConTEB's training splits plus NQ evidence spans. Evaluation is out of distribution:

  • ConTEB's COVID-QA, ESG Reports, Football, Geography, and Insurance splits. EBCAR's ablation shows Insurance depends heavily on intra-document position, which makes it a likely place for boundary effects.
  • DAPR's five corpora, including NQ-hard, Genomics, and ConditionalQA.
  • Downstream QA on the PICWiki suite, using the generator from PIC.

Primary diagnostic. The primary diagnostic is the evidence-split rate: the fraction of gold evidence regions cut by at least one chunk boundary. It directly measures fragmentation, independent of embedder quality. Retrieval is reported as evidence recall at a fixed token budget and as nDCG@10. Relevance for variable-length chunks is defined by overlap with gold evidence, applied identically to all chunkers.

Decisive checks. Three outcomes would decide the claim:

  1. The full model must lower the evidence-split rate relative to both PIC and the untrained dynamic program, and must convert that into budget-matched recall gains on the out-of-distribution ConTEB domains. A lower split rate without recall gains would show that splits are not the bottleneck.
  2. Placing our chunks under EBCAR must raise EBCAR's ConTEB score. That would show boundary repair and cross-chunk reranking address different errors.
  3. Indexing cost, measured as wall-clock time per million tokens on one A100, must stay within a small constant of Late Chunking. The added dynamic program is $O(S \cdot D)$ on top of an encoder pass we already pay for.