Letting a Long Document Supply Its Own Dictionary for Chinese Word Segmentation

Chinese, Japanese and other scripts written without spaces must be split into words before most downstream processing can happen. The oldest reliable rule for this is "longest match": when several dictionary words compete for the same characters, take the longest one. The rule works well until the text leaves the domain its dictionary was built for. In a web novel, a patent or a medical text, the important words are new coinages such as character names, technique names, compounds and drug terms. Each of these usually contains a shorter, ordinary word. Greedy longest match grabs that shorter word and breaks the real unit apart. Neural segmenters trained on newswire have the same failure in a milder form, and they are especially bad at splitting words five or more characters long.

This proposal starts from an observation about long documents. The words a newswire model does not know are exactly the ones a long document repeats: a protagonist's name, a patented compound, a disease term. Most systems cut a long document into overlapping windows and segment each window on its own. That throws away the most useful domain evidence available, which is that the same character string recurs, with stable edges, across dozens of windows. We propose building a small lexicon for each document at inference time from these recurrences. Candidate strings are scored by how often they recur and by how confidently a pretrained segmenter places boundaries at their edges across all occurrences. Each window is then decoded with a length-weighted longest-match bonus for entries in this lexicon. Once the full novel compound is in the lexicon, the longest-match rule prefers the whole word to its shorter sub-word, so the rule's main weakness becomes its strength. Because every window consults the same lexicon, windows also stop disagreeing about how to segment the same string.

Core Claim

Document-scoped lexicon induction, used as a soft, length-aware longest-match prior on top of a newswire-trained neural segmenter, should recover most of the out-of-domain word-boundary errors on long cross-domain texts without any labeled or unlabeled target-domain corpus. Three measurable effects follow from this. First, recall on out-of-vocabulary words and on words of five or more characters should rise substantially on long documents such as novels and patents. Second, the gain should shrink toward zero on short, non-repetitive text such as microblogs, and the method should not hurt there. Third, the gain should depend on evidence from the specific document and not only on its genre, which we test by swapping in the lexicon induced from a different book of the same genre.

Most of the work at this intersection handles only two of these ingredients at once. On longest-match brittleness, Zong et al. identify that greedy matching commits to a shorter dictionary entry inside an out-of-vocabulary compound. This is the failure we target, and it motivates the proposal. Wang et al. score candidate segmentations by squared word length plus frequency so that long matches win over chains of short ones. We reuse their length weighting, but apply it to a lexicon that is induced per document instead of a static one. On domain shift, WEB-CWS trains word embeddings on unlabeled target-domain text and decodes by cosine similarity. It raised precision on "glucocorticoid" from 3.8% to 76.9%, which shows that target-text statistics can recover domain words. It needs a target corpus, however, and uses no length prior. The cascaded Weibo CRF repairs out-of-vocabulary fragments using a 278,000-entry external word bank and a character-cohesion statistic, working one post at a time. We borrow cohesion as one feature, but compute it across the occurrences of a string within a document instead of within one sentence.

On long-document processing, the OFLS study (filed under the InstructCoder title) shows that fixed-length windows with 50% overlap beat sentence-based splitting for alignment. That line of work treats context preservation as keeping enough neighbors inside each window. Our proposal instead treats it as carrying lexical evidence across all windows.

Why we did not pursue the obvious idea

The research file's first gap suggests aligning window cut points to longest-match word boundaries so that windows never slice a word in half. We dropped this idea. In Chinese prose, cutting at sentence-final punctuation, or running 50%-overlap windows and keeping each window's predictions only from its central region, already ensures that almost every word is seen whole in some window. Either trick is simpler than boundary-aligned seams and would plausibly perform just as well. The errors that remain on novels and patents are not caused by cut points. They occur because the segmenter does not know the word at all, even when the word appears in full, with context on both sides. Seam alignment does not address that. We keep punctuation and center-stitched cutting as the windowing layer in every system, including baselines, so that seam effects are held constant and do not confound the comparison.

Research Program

The base segmenter is a character-level pretrained encoder with a boundary tagging head, fine-tuned on newswire only (CTB and PKU standards). A ZEN-style n-gram-enhanced encoder serves as a second backbone, to test whether a static n-gram lexicon already captures the effect. A document is processed in center-stitched overlapping windows.

Lexicon induction has three steps. We enumerate every substring of 2 to 10 characters that recurs at least $k$ times in the document, which a suffix array does in near-linear time. For each candidate we compute three statistics across its occurrences: the recurrence count, the variety of characters at its left and right edges (a string that is a real word should be followed and preceded by many different characters), and the segmenter's average probability of a boundary at both edges and of no boundary inside. Candidates that are strict substrings of a higher-scoring candidate, and always appear inside it, are discarded. This step specifically removes the "short word hidden inside a novel compound" pattern.

Decoding runs over the lattice of possible segmentations of each window and maximizes

$$\sum_{w \in \text{seg}} \Big[ \log p_\theta(w \mid \text{window}) + \lambda \, s_D(w) \, |w|^{\alpha} \, \mathbb{1}[w \in L_D] \Big],$$

where $p_\theta(w \mid \text{window})$ is the segmenter's probability of word $w$ (a product of its per-character boundary probabilities), $L_D$ is the lexicon induced for document $D$, $s_D(w) \in [0,1]$ is the combined evidence score of $w$ in that document, $|w|$ is the length of $w$ in characters, and $\lambda$ and $\alpha$ are scalars. Setting $\alpha = 2$ recovers the squared-length weighting of Wang et al. We tune $\alpha$, $\lambda$ and $k$ on newswire development data plus one held-out novel, never on the test domains. We then run one refinement pass: the edge probabilities in $s_D$ are recomputed from the first-pass segmentation, so the segmenter's in-context decisions and the document statistics correct each other.

As a secondary application, we test whether the same prior reduces long-entity fragmentation in Chinese named entity recognition, the task of tagging names of people, places and organizations. Partial boundary cuts are known to corrupt name taggers (Ji et al.; Chen & Kong). Here the prior is added as a span feature, similar to the binary lexicon-membership features of Neural Lexicons for Slot Tagging. That paper found such features worth nearly 12 absolute F1 points in low-resource settings (40.01% to 52.78%), but its lexicons were static and hand-built.

Baselines

The naive reference points are the newswire-trained segmenter applied window by window with no lexicon, and forward maximum matching (greedy longest match) over a static general dictionary. The second shows how badly classic longest match breaks under domain shift. Both use the same center-stitched windows.

The strongest existing method is the teacher-student approach of Neural Regularized Domain Adaptation. A newswire teacher regularizes a student trained on a small labeled target set. It reports 94.27 F1 on the Zhuxian novel domain (against an 85.48 baseline) and 95.69 F1 on patents (against 86.31). It uses labeled target data and we use none. We therefore treat its numbers as a supervised ceiling. We also rerun it with the same encoder backbone and a matched fine-tuning budget, and report both it and the combination of it with our prior. Beating it outright is not required. The result we care about is closing most of the gap between the 85.48-style unadapted baseline and the supervised ceiling without target labels, plus any additive gain when the prior is stacked on the supervised model.

The strongest label-free competitor is WEB-CWS. It reports gains of more than 3.0% word F-measure on four of five domains, but the file gives no absolute scores for it. We rerun it under two conditions: with its unlabeled target corpus (the whole genre), and with only the test document as training text. The second condition matches our information budget exactly.

The ablations isolate the key ingredient. Setting $\alpha = 0$ keeps document recurrence but removes the length preference. Replacing $L_D$ with a static domain dictionary keeps the length preference but removes document induction. The genre-swap control uses $L_{D'}$ from a different novel of the same genre. A post-hoc consistency pass forces every occurrence of a repeated string to take its majority segmentation. Finally, a corpus-level lexicon induced from all test-genre text isolates document-scoped induction from genre-scoped induction.

Why not just…?

Why not just prompt a frontier LLM on each window? The historical Chinese comparison found that GPT-4o, Claude 3.5 Sonnet and o3-mini beat Jieba and spaCy on Republican-era segmentation, so accuracy is a real threat. The same study reports that these models need far more time and tokens, with o3-mini the slowest, and prompted windows would still be decoded independently of one another. We will run GPT-4o per window on a fixed subsample of each test domain, reporting both F1 and cost per thousand characters. We will also test feeding our induced lexicon to the prompt, which checks whether document evidence helps LLMs too. If the LLM wins on accuracy, our claim narrows to accuracy at a given cost, and we will say so.

Why not just self-train the segmenter on its own predictions for the document? Test-time self-training is the most serious alternative. It uses the same information as our method, and in principle it could learn the recurring compounds. Its known risk is reinforcing its own mistakes: a fractured compound that is predicted consistently becomes a confident pseudo-label. We run it with the same per-document compute budget. The decisive comparison is recall on words of five or more characters that the base model initially splits. If self-training matches our method there, the explicit length-weighted lexicon is unnecessary, and we would report that honestly.

Why not just enforce consistency across repeated strings after segmentation? Majority voting over occurrences is cheap and fixes disagreements between windows, but it only picks among segmentations the model already favors. If the base model splits a novel name on most occurrences, voting makes that split universal. Our method differs because the length term can overturn a majority split when the edge statistics support the longer unit. The majority-vote ablation measures exactly this difference.

Why not a lexicon-free character model with enough capacity? CAN-NER argues that external lexicons are unnecessary, yet its authors concede weaker OntoNotes 4 results than lexicon-using models. The ELMo segmenter reaches 95.5% out-of-vocabulary accuracy on UD Chinese but still fractures entities longer than five characters. Our base model already is a strong character model, so every reported gain is measured over this alternative.

Proposed Experiments

The primary testbed is the five cross-domain sets used by WEB-CWS: the novels DouLuoDaLu, FanRenXiuXianZhuan and ZhuXian, plus Dermatology and Patent. We add the Zhuxian and patent splits used by Neural Regularized Domain Adaptation, and the CTB9 weblog, SMS/chat and conversational genres. Where labeled sentences come from a longer source text, the lexicon is induced from the full unlabeled source, and scores are computed only on the labeled sentences. The SIGHAN 2012 Weibo set serves as a short-text negative control. There the reference is the cascaded CRF's reported peak of 0.933 F1, and our requirement is no degradation. We will also run the method on the Republican-era passages, while noting that 208 passages give little recurrence.

Metrics are word F1, out-of-vocabulary recall, recall on words of five or more characters, and a consistency rate: the fraction of recurring gold words segmented identically across all their occurrences. For the NER extension we use OntoNotes V4 and Weibo NER. There the reference is the boundary-focused model's 79.95% and 70.14% F1 (Chen & Kong), and we additionally report long-entity recall.

Three checks decide the question. First, on the novels and patents, the method should raise long-word recall over the unadapted segmenter by a margin that clearly exceeds both self-training and majority voting under matched compute. Second, the genre-swap control should lose most of that gain; if it does not, the effect comes from genre vocabulary and a corpus-level lexicon would suffice. Third, the gain should grow with how often out-of-vocabulary words recur in a document and should vanish on Weibo. If all three hold, the result shows that the dictionary a long document induces for itself turns longest match from a liability under domain shift into a usable prior.