Do Over-Split Words Leak Context Out of Late-Chunked Embeddings?

Retrieval systems cut long documents into chunks so that each chunk fits a model and can be matched to a query. Cutting loses context: a chunk that says "it" no longer knows what "it" refers to. The current best inexpensive fix is "late chunking." The whole document is run through a long-context encoder first, and only afterward are the per-token vectors averaged into chunk vectors, so each chunk carries information from the rest of the document. These systems treat the tokenizer, the component that breaks words into subword pieces, as a fixed given. This note asks what happens when the tokenizer breaks a rare technical or morphologically complex word into many meaningless pieces, as "demon" might become "de" + "mon." Our hypothesis is that such over-split words hurt late chunking in two ways. Their fragment vectors carry less of the document context. And plain averaging lets a word split into six pieces count six times in the chunk vector, while a word kept whole counts once.

The proposal has two parts. The first is a controlled causal test of whether word-level over-segmentation and chunk-level fragmentation interact. The second is a training-free repair called unit-aware late pooling. It first averages token vectors within linguistically valid units, meaning morphemes found by a segmenter designed to avoid spurious splits, and only then averages those units into a chunk vector. If the interaction is real, the payoff is largest exactly where retrieval matters most: scientific, medical, and technical queries whose key term is a rare compound the tokenizer shreds.

Core Claim

Late chunking (Günther et al.) reports a relative nDCG@10 gain of about 3.6% over naive chunking on BeIR. We claim that this gain is not uniform. It should shrink, or reverse, for chunks whose query-relevant terms have high tokenizer fertility, meaning many subword pieces per word, and especially when many of those piece boundaries fall at non-morphological positions. We further claim that part of this loss is a pooling artifact, not an encoder failure, so it can be recovered without retraining. Recovery comes from pooling over morpheme-level units rather than raw tokens.

The research file's own gap analysis notes that neither late chunking nor Landmark Embedding (Luo et al.) examines how subword over-segmentation shapes the token states they pool. Landmark Embedding inserts boundary marker tokens into a single encoder pass and pools at those markers. We found no work in the file that measures this interaction or changes the pooling unit in response to it.

How This Builds on Each Direction

Morphological over-segmentation is the mechanism and supplies the tools. Snover et al. characterise the failure: statistical prefix extraction cuts coincidental strings such as de+mon. Manning argues that boundary errors at this level propagate into downstream systems. Neither studies modern subword tokenizers inside dense retrievers, and that is the step we take.

To decide which tokenizer splits are spurious, we reuse two segmenters built to suppress over-segmentation. The first is the morphological forest of Luo et al.. It links each word to its morphological parent and uses a global penalty on the number of distinct affixes, solved by integer programming, to prune fake affixes. The paper reports a 27.7% relative F1 gain in German, and English word-family clustering precision rising from 0.328 to 0.895. The second is MORSE (Sakakini et al.), which accepts a split only if word-vector geometry supports it. It reports 81.29 F1 against Morfessor's 57.60 on its SD17 benchmark. MORSE's documented weakness on sparse agglutinative vocabularies such as Finnish limits where we can trust it. For that reason the forest segmenter is our primary unit source, and Morfessor 2.0 (Smit et al.) serves as a deliberately over-splitting control.

Contextual fragmentation is the application regime. Late chunking motivates the work and supplies the base method. The "stitched illusion" result of Qu et al. constrains the design. They find that smarter boundary placement rarely beats fixed-size chunks on natural text: fixed-size chunking reaches 90.59% F1@5 on HotpotQA. We therefore deliberately leave chunk boundaries alone and change only what happens inside a chunk. Any gain we report cannot be another variant of semantic boundary placement, which Qu et al. largely refute.

Research Program

The first step is measurement. For the tokenizer of a long-context encoder of the kind used in the late chunking paper, we score each word by two quantities. The first is fertility. The second is the spurious-boundary rate: the share of its token boundaries that do not coincide with morpheme boundaries. We validate the boundary judgments against Morpho Challenge 2010 gold segmentations in English, German, Turkish, and Finnish, and against CELEX morphological structure. This tells us how much to trust the forest segmenter as a proxy on unannotated retrieval corpora.

The second step is a causal test. On a fixed corpus, we artificially raise fragmentation at inference time by forcing finer splits on a random subset of words, at controlled rates. We cross this with the chunking condition: naive chunking versus late chunking. The 2×2 design measures retrieval quality and the cosine drift of chunk vectors relative to the unperturbed run. A positive interaction means late chunking's gain falls faster under fragmentation than naive chunking's score does. That would show over-segmentation leaks context beyond a uniform penalty. To separate the encoder effect from the pooling effect, we also log how much each fragment's state changes between full-document and isolated-chunk encoding. This is a direct measure of how much context each fragment absorbs.

The third step is the repair. Let a chunk contain units $U$, where each unit $u$ is a set of tokens forming one morpheme according to the forest segmenter, and let $h_t$ be the contextualised vector of token $t$ from the full-document pass. Unit-aware late pooling computes

$$c = \frac{1}{|U|}\sum_{u \in U} \frac{1}{|u|}\sum_{t \in u} h_t ,$$

so each morpheme contributes once, regardless of how many pieces the tokenizer cut it into. Standard late chunking is the special case where every token is its own unit. We also test word-level units, where whitespace words are pooled as one, to isolate whether the morphological segmentation adds anything beyond undoing fertility weighting.

Baselines

The do-nothing reference is naive chunking: fixed-size chunks encoded independently and mean-pooled, using the same encoder and chunk size. Following Qu et al., fixed-size chunking is a hard reference on natural text, not a strawman.

The strongest existing method is standard late chunking as described by Günther et al. It is run with the same encoder, the same 8,192-token window, and identical chunk boundaries, so the only difference between it and our method is the pooling unit. No new hyperparameters are tuned for our method beyond the segmenter, which is fixed after Morpho Challenge validation. The only figure the file reports is the roughly 3.6% relative nDCG@10 improvement over naive chunking on BeIR; no absolute nDCG@10 for late chunking on any single BeIR subset is reported on Lacuna, so we reproduce it ourselves. Beating late chunking means a statistically significant nDCG@10 gain on the high-fertility query stratum, with no loss on the low-fertility stratum.

The key ablation is word-level pooling, which removes the morphology. If it matches morpheme-level pooling, our contribution reduces to "count each word once." We would report that honestly. A second ablation pools over Morfessor 2.0 units. Because Morfessor over-splits, it should perform between token-level and forest-level pooling if the spurious-boundary account is right.

A sparse-plus-dense hybrid, which combines BM25 keyword scores with late-chunking scores, is run as an additional competitor for the reason given below.

Why not just…?

Why not just add BM25 keyword retrieval, which handles rare terms exactly? This is the most serious threat. Rare, heavily fragmented terms are precisely where exact keyword match is strong. If a hybrid of BM25 and late chunking closes the high-fertility gap entirely, our repair has no practical niche. We therefore evaluate unit-aware pooling both alone and inside the hybrid. The claim survives only if it improves the hybrid on the high-fertility stratum, or wins where keyword matching fails: queries that paraphrase the rare term or use a different inflection of it.

Why not just pool at the word level and skip morphology? It is cheaper and might suffice. The forest results of Luo et al. suggest that compounds and derivations carry separable meaning-bearing parts that whole-word pooling would merge. This argument is plausible but not established for retrieval. The word-level ablation decides it directly, and we commit to reporting a tie.

Why not just extend the tokenizer vocabulary with domain terms or fine-tune the encoder? This would remove the cause rather than patch the pooling, but it requires retraining for each domain and language. Pre-training mismatches of the kind reported for QDS-Transformer (Jiang et al.) warn that architectural changes to pre-trained encoders are not free. We treat this as an upper bound on one BeIR subset rather than a competitor across all of them, and report how much of its gain the training-free pooling recovers.

Proposed Experiments

Retrieval is evaluated on BeIR subsets with dense technical vocabulary: SciFact, NFCorpus, and TREC-COVID, with FiQA as a lower-jargon contrast. Queries are stratified by the maximum fertility of their gold-relevant terms. Evidence selection on QASPER (Dasigi et al.) tests the multi-paragraph setting. There the reported LED evidence-selection F1 is 39.37 against 71.62 for humans, and 55.5% of questions need evidence from multiple paragraphs. The Chinese portion of LongBench (Bai et al.) probes a writing system where compound-internal boundaries are the known failure mode of entropy-based segmentation (Jin et al.). For the Chinese setting, units come from a branching-entropy segmenter rather than the forest.

Three results would be decisive. The first is the interaction term in the 2×2 forced-fragmentation design. If it is zero, the core claim is false, and we would publish the measurement as a negative result. The second is whether late chunking's per-query gain over naive chunking correlates negatively with the spurious-boundary rate of the query-critical terms on natural BeIR queries. The third is whether unit-aware pooling recovers the high-fertility gap more than word-level pooling does, and whether it still helps inside the BM25 hybrid. Only if all three hold does the proposal deliver what it claims: a measured link between two kinds of fragmentation and a retraining-free fix for it.