Topics as Cheap Document Context for Low-Resource Text Normalization

Text normalization rewrites messy spellings into a standard form: historical orthography, OCR errors, dialect. When labeled examples are scarce, the best current systems pair a character-level rewriting model with a language model of the standard language and train both on unlabeled text. Those language models only look at the surrounding sentence. Whether an ambiguous historical or OCR-damaged word should become "ship" or "shop", "mass" or "mess", often depends on what the whole document is about. That is exactly the information a topic model extracts, and it extracts it robustly from noisy text. We propose feeding document-level topic mixtures back into the normalizer's language model and alternating the two until they agree. Each topic mixture is inferred once per document and carried across every window of a long text. The document therefore gets one consistent thematic prior, and it costs only a handful of numbers rather than a data-hungry long-context neural model.

The proposal rests on an observation that has so far been treated as a curiosity. Topic Modeling on Historical Newspapers found that plain LDA recovers coherent historical themes from uncorrected OCR text, and that skipping spelling correction barely hurt. If topics survive noise that the normalizer is struggling with, topics are a signal the normalizer can use. Existing pipelines never exploit this. They normalize first and topic-model afterwards, as in the Slovenian periodicals giga-corpus (A Lightweight Approach to a Giga-Corpus of Historical Periodicals), so the flow of information runs one way only.

Core Claim

In low-resource normalization, a document-level topic prior reduces errors on the tokens that sentence context cannot resolve: ambiguous variants and first occurrences of a variant in a document. It does so beyond what a simpler per-document consistency cache achieves. It also stays usable on long documents, because the topic mixture works like a persistent header shared by all overlapping windows. A second, downstream claim follows. Topics re-estimated on the jointly normalized text will be more coherent and less fragmented than topics built on raw text or on pipeline-normalized text.

The claim is deliberately stated against a cheap competitor. A reviewer will point out that "one normalization per variant per document" might capture most of the document-level gain. We therefore build that cache into the model as its own component and treat the topic term's gain over the cache as the quantity that has to be positive.

Mechanism

We start from the semi-supervised noisy-channel normalizer of Semi-supervised Contextual Historical Text Normalization. A noisy-channel model scores a candidate normalization by combining two things: how plausible the candidate is as modern text, and how likely it is to have been garbled into the observed historical form. That paper does this with a character-level neural transducer $p(x_i \mid y_i)$, which maps modern words $y_i$ to observed historical forms $x_i$. It pairs the transducer with a sentence-level modern language model $p(y)$ and trains the transducer by Expectation-Maximization on unlabeled historical text. Expectation-Maximization is an iterative procedure that alternates between guessing the hidden normalizations and re-fitting the model to those guesses. We keep the transducer and the training procedure unchanged, and replace the prior for document $d$ with an interpolation:

$$ p(y_i \mid y_{<i}, d) = (1-\lambda-\mu)\, p_{\text{LM}}(y_i \mid y_{<i}) + \lambda \sum_{k} \theta_{d,k}\, \phi_k(y_i) + \mu\, c_d(y_i) $$

Here $p_{\text{LM}}$ is the original local language model. $\theta_{d,k}$ is document $d$'s weight on topic $k$, and $\phi_k$ is topic $k$'s distribution over modern words. $c_d$ is a cache distribution over modern forms already chosen elsewhere in the same document. The interpolation weights $\lambda$ and $\mu$ are tuned on the small labeled set.

Training alternates three steps. First, fit LDA on the raw unnormalized corpus to initialize $\theta_d$, relying on LDA's documented tolerance to noise. Second, decode normalizations with the interpolated prior. Third, refit $\phi_k$ and $\theta_d$ on the expected modern forms from the decoding step. Moving the topic–word distributions onto the modern vocabulary is what makes the loop bidirectional. It also means spelling variants of one word stop splitting a topic's probability mass among themselves.

Segmentation enters where documents exceed what the local model or transducer can handle. Long periodical issues are decoded in overlapping windows, using the 512-unit window and 128-unit slide from Exploring the Power of Romanian BERT for Dialect Identification as the default and the 50% overlap reported as a good trade-off in InstructCoder as an alternative. The $\theta_d$ and $c_d$ inferred for the whole document are shared by every window. This mirrors the persistent context header that Leveraging Pretrained Models for Automatic Summarization of Doctor-Patient Conversations prepends to each chunk, except that our header is a compact latent state rather than text. When overlapping windows disagree, the reconciliation rule keeps the hypothesis from the window in which the token is most central. The file does not supply the re-segmentation problem that Makarov & Clematide leave open (aller handt → allerhand), so we treat it as out of scope and report split and merge cases separately rather than claiming to solve them.

Research Program

The work has three parts. The first builds the joint normalizer on top of the released noisy-channel design and implements the cache and topic components so each can be switched off independently. The second is an error analysis that stratifies every test token into four groups: unambiguous, ambiguous-within-type, first occurrence in a document, and repeat occurrence. This lets us attribute gains to the component that should produce them. If the topic term helps, the gains should concentrate on ambiguous first occurrences, and the cache should dominate on repeats. The third part measures what the joint model does to downstream topic quality: coherence, and the rate of topics whose top words are spelling variants of each other. Here the proposal speaks to the fragmentation problem that BERTopic for Topic Modeling of Hindi Short Texts and NGLUEni attribute to unnormalized, morphologically rich text.

Baselines

The naive reference point is the identity baseline, which leaves every token as it is. A Multilingual Evaluation of Three Spelling Normalisation Methods for Historical Text reports this at 75.8% accuracy and 0.26 character error rate for historical English. We also include character-level statistical machine translation, the tool family that dominates small-data settings. That paper reports 94.3% accuracy and 0.07 character error rate on English with a GIZA unigram model. On dialect tweets, Murreviikko reports chrF2 of 84.4 for Moses, ahead of ByT5 at 83.6. On OCR'd Slovenian periodicals, CSMTiser reduced word error rate from 5.4% to 4.4% (Erjavec et al.). All of these systems are retrained on exactly our labeled budgets.

The strongest competitor is the semi-supervised noisy-channel system of Makarov & Clematide. It matters most because our model differs from it only in the prior. It reports an average 19% error reduction over prior systems with 500 labeled tokens, 88.4% of supervised accuracy with zero labels, and a 38% error reduction on ambiguous German tokens from its contextual prior. The file gives no per-language absolute accuracies for this system (no reported result found on Lacuna for per-language accuracy), so we rerun it ourselves. It gets the same eight datasets, the same 500-token and zero-label budgets, the same unlabeled corpora, the same modern Wikipedia-style reference text, and the same number of EM iterations and hyperparameter trials. Beating it means a statistically significant reduction in word error, measured by paired bootstrap over documents, on a majority of languages at the 500-token budget. It also means a larger reduction on the ambiguous-first-occurrence stratum than on unambiguous tokens.

Two ablations isolate the key ingredient. Cache-only sets $\lambda = 0$. Topic-only sets $\mu = 0$. A further ablation freezes topics at their raw-text initialization instead of refitting them, which tests whether the feedback loop matters or a one-shot topic prior is enough. Finally, a ByT5 baseline decodes each window with the preceding window's text prepended. This is the brute-force way to add document context, and it gets the same labeled data and a matched fine-tuning budget.

Why not just…?

Why not just use a document cache, "one normalization per variant per document"? This is the most serious threat, and it may well capture most of the document-level gain on repeated variants. We do not argue around it. It is a component of our model and a standalone ablation. A cache cannot help the first occurrence of a variant, and it cannot help when the same surface form legitimately maps to different words within one document. The stratified analysis measures exactly these cases. If topic-plus-cache does not beat cache-only on first occurrences, we will report that the cache suffices and that topics contribute only on the downstream coherence side.

Why not just give a neural model a longer context window? MuLD shows that native long-context models beat chunked ones when they can be trained. In our regime, though, the evidence points the other way. With modest data, ByT5 does not beat character-level statistical translation on Murreviikko. Recording for Eyes, Not Echoing to Ears found that feeding whole documents to large models caused drift and hallucination, and that gains required chunking plus explicit keyword anchors. Our topic mixture plays a role close to those anchors, but it is learned without supervision. The context-prepended ByT5 baseline settles the question directly on equal data.

Why not just normalize first and then topic-model, or skip normalization since LDA is noise-robust? The pipeline route is what the Slovenian corpus did. It is our "topics frozen after normalization" condition and our downstream comparison. The noise-robustness result from Yang et al. concerns coarse themes in a large English archive. It says nothing about variant-split topics in small or morphologically rich corpora. We measure variant fragmentation and coherence on raw text, on pipeline-normalized text, and on jointly normalized text.

Proposed Experiments

The primary testbed is the eight historical datasets from Makarov & Clematide, evaluated at 0, 500, and a few thousand labeled tokens. A necessary first check is whether document boundaries are recoverable for each dataset. The file does not state this, so any dataset without them is dropped from the topic experiments rather than given pseudo-documents silently. We supplement these with the five-language corpora of Pettersson et al., measuring accuracy and character error rate.

The long-document condition uses the Slovenian periodicals together with the 300-paragraph CSMTiser supervision. Here we measure word and character error rates, and we compare against the 4.4% and 1.0% CSMTiser figures from Erjavec et al.. This condition also varies the window overlap, to test whether sharing $\theta_d$ across windows makes accuracy insensitive to where token boundaries fall.

As a stress test, Murreviikko groups tweets by author or thread into pseudo-documents and reports chrF2. We expect small or no gains here, because tweets are short and topics are weak. That outcome is itself informative about how much document length the method needs.

Downstream, we fit LDA and embedding-based topic models on each normalization output of the Slovenian corpus. We report C_v and NPMI coherence against the ParlaMint reference, together with the variant-duplicate rate.

The decisive checks are three. The first is whether topic-plus-cache beats cache-only on ambiguous first occurrences at the 500-token budget. The second is whether refitting topics on normalized text beats frozen raw-text topics. The third is whether the gains survive against context-prepended ByT5 under equal data. The first check is the one the proposal stands or falls on.