Where in the Window Does Language Drift? Offset-Resolved Adaptation for Historical Tagging

A tagger that labels each word from a small window of its neighbors (say two words to the left and two to the right) has to rely on those neighbors staying useful. When the tagger is moved from modern news to seventeenth-century prose, some neighbors stop helping. A word's own spelling changes. A preposition one slot to its left changes role. A verb that used to sit next to its subject has moved to the end of the clause, far outside the window. Current adaptation methods treat all of this as one undifferentiated domain shift. They either normalize spellings or refit the whole model. We propose to take the window apart position by position. The tagger keeps one shared set of weights per relative offset (the target word, one to the left, three to the right, and so on) and adds a small era-specific correction for each offset. A penalty keeps a correction at exactly zero unless the historical data demands it.

With a few thousand labeled historical tokens, this should do two things. It should adapt more sample-efficiently than refitting everything, because it spends the target data only where the context actually changed. It should also produce a readable map of which positions drifted in which era. That map is a claim about language change which can be checked against independent syntactic evidence, such as the long-distance, clause-final dependencies that make historical German parsing collapse (Nissim and Salomoni).

Core Claim

Temporal drift is not uniform across the context window. Its location is identifiable from little labeled data, and adapting only the drifted offsets beats both whole-model refitting and a fixed window at small target budgets. Concretely, the model's weights for the features at offset $o$ in era $e$ are written as

$$w_{o,e} = w_o + \delta_{o,e}, \qquad \text{penalty} = \lambda \sum_{o,e} \lVert \delta_{o,e} \rVert_2 .$$

Here $w_o$ is the weight block shared across eras for all features at offset $o$ (tokens, affixes, word shapes), and $\delta_{o,e}$ is the era-specific deviation. The penalty is a group lasso: it sums the unsquared lengths of whole blocks, so it drives entire offset blocks to zero rather than individual weights. The set of offsets with nonzero $\delta_{o,e}$ is the drift map. The window is deliberately wide (up to $k=6$), so the model can express the case where a governing element has moved outside the usual radius. Without the group penalty, the wide window can learn to use distant positions but gives no map of which ones changed.

Relation to Prior Work

The closest work is Part-of-Speech Tagging for Historical English. It embeds entire feature templates, including local context templates, and conditions them on year and genre (FEMA-attribute). This lifts Penn-Treebank-trained tagging of Early Modern English from 74.15% to 77.92%, and to 79.05% with spelling normalization. FEMA is unsupervised, and its window geometry is fixed and hand-set. It shows that local context carries the disambiguation signal, but not where drift sits inside the window. We keep its feature templates and replace the fixed geometry with per-offset, era-specific corrections learned from a small labeled sample.

Three papers motivate the gap by showing that local taggers fail in specific positional ways. Automatic Verb Extraction from Historical Swedish Texts reports that its trigram HMM tagger breaks when historical syntax puts governing elements in unseen positions. It also shows that adding historical lexicons can hurt, which argues against fixing drift purely at the word level. Nissim and Salomoni report that on historical German, POS tagging holds at 97.3% while attachment score falls from 84.6% to 65.4%. They attribute the fall to long-distance and verb-final relations. This predicts that drift for some tags should appear at far-right offsets. A Named Entity Recognition Shootout for German finds that CRFs with window templates beat target-only BiLSTMs on small historical newspapers (73.31 vs 70.46 F1), while transfer learning reaches 78.56. That result frames the regime we target: small labeled historical sets, where inductive bias matters.

Two papers supply the method. Pointwise Prediction for Robust, Adaptable Japanese Morphological Analysis shows that per-position logistic regression over wide character windows matches or beats a joint CRF (98.03% vs 97.31% in domain, 95.13% vs 94.57% on web text) and adapts with 1,500 partially annotated words. We adopt its pointwise, independently trained classifier, which makes offset blocks cleanly separable. A Robust Risk Minimization based NER System shows that affix features at local positions drive most of the gains (dev F1 82.15 to 86.07), so our offset blocks include affixes and not only word identity.

One paper constrains how much the method can claim. Where large historical pretraining corpora exist, D'AlemBERT reaches 95.60% on Early Modern French, against 89.27% for modern CamemBERT. We do not expect a windowed linear model to beat such a model where one can be built. That comparison bounds our claim to eras and languages without pretraining corpora of that size, and to the offset map itself.

Research Program

We first build a pointwise tagger (multiclass logistic regression) over offsets $-6$ to $+6$. Each offset contributes its own feature block: token identity, prefixes and suffixes up to length 4, character shape, and capitalization. The tagger is trained on modern source data.

On each historical era we then fit the shared-plus-deviation model with the group penalty, using target samples of 500, 1k, 2k, 5k and 10k labeled tokens. The penalty $\lambda$ is selected on a held-out historical development split.

For each era we report the drift map: which offsets have nonzero corrections, how large each correction is, and which tag classes it mainly affects. We also report how the map changes as $\lambda$ varies, and how stable it is across bootstrap resamples of the target data.

A second variant replaces the discrete feature blocks with frozen contextual vectors from a modern pretrained encoder, gathered at each offset. This tests whether the offset structure still matters when each position already carries sentence-level context.

Finally, we test whether the map is linguistic rather than statistical noise. We compare the offsets flagged for verbs and their arguments against dependency-distance statistics measured on the same historical texts. We also check whether maps learned on Early Modern English carry over, in direction, to the later Modern British English period.

Baselines

The naive reference point is the source-trained window tagger at the conventional radius $k=2$, applied to historical text with no adaptation. On Early Modern English the comparable reported figure is 74.15% accuracy (Eisenstein et al.). We rerun it on our splits rather than quoting it. A second reference applies VARD-style spelling normalization before this tagger, so we can see how much of the gap is purely orthographic.

The strongest existing method on this benchmark is FEMA-attribute with normalization, reported at 79.05% on PPCEME in the same paper. FEMA is unsupervised, so a raw comparison would be unfair to it. We therefore also give FEMA the same labeled target tokens, appended as extra supervised training data, and run both systems with the same feature templates and the same number of hyperparameter trials. Beating it means higher accuracy at every target budget up to 5k tokens, and specifically on out-of-vocabulary tokens, where FEMA reaches 63.16%.

For NER on the Austrian newspaper corpus (ONB), the strongest reported system is the BiLSTM with GermEval transfer at 78.56 F1 (Riedl and Padó). For Early Modern French the reference is D'AlemBERT at 95.60% (Gabay et al.). For both we rerun a modern pretrained encoder fine-tuned on each target budget, with a matched tuning budget.

Two ablations isolate the key ingredient. The first uses the same wide window with an unstructured era-specific correction on every weight, penalized by ordinary L2 shrinkage. This is feature-augmentation domain adaptation without offset grouping. The second is the grouped model restricted to $k=2$. If the first ablation matches the full model, the grouping adds nothing beyond interpretability. If the second matches it, the wide window adds nothing.

Why not just…?

Why not just normalize spelling and tag with a modern model? Normalization repairs the target word's own offset and nothing else. Eisenstein et al. report that normalization combined with feature embeddings still leaves Early Modern English near 79%, far below the 97% seen on modern text. They attribute the residual to syntactic constructions such as "drew unto him." Pettersson and Nivre find that their normalized pipeline plateaus at 0.85 F-score. Our normalization baseline measures this directly. The prediction is that after normalization, the drift map loses its offset-0 correction but keeps its non-zero offsets. If instead all correction disappears after normalization, the proposal is wrong and we will say so.

Why not just fine-tune a pretrained transformer on the small historical set? This is the real threat wherever a good pretrained encoder exists. Transfer closes the NER gap on ONB (78.56 F1), and D'AlemBERT dominates on Early Modern French. We do not argue around this. The budget-curve experiment reports where fine-tuning overtakes the grouped model, and the contextual-vector variant tests whether offset-grouped corrections still help on top of a frozen encoder. We claim an advantage in two places only: the smallest target budgets, and eras without a pretraining corpus of D'AlemBERT's 185M-token scale. The drift map is a separate deliverable that fine-tuning does not produce.

Why not just widen the window and refit everything with L2 regularization? This could plausibly match our accuracy, which is why it is the central ablation rather than an afterthought. If it does match, the accuracy claim collapses to "use a wider window." The remaining contribution would then be the map. It must justify itself through the validation against dependency distances and its stability across resamples, and we would reframe the work as a measurement study of drift.

Proposed Experiments

The main benchmark is POS tagging from the Penn Treebank to PPCEME and PPCMBE, following the tag mapping of Eisenstein et al. so that annotation drift between treebanks is handled the same way for every system. Secondary benchmarks are Early Modern French POS on FREEM$_{LPM}$ and historical German NER on ONB. For each we report accuracy or F1 against the number of target tokens, out-of-vocabulary accuracy, and per-offset correction norms with bootstrap confidence intervals.

The first decisive check is the budget curve. The grouped wide model should beat both the unstructured-correction ablation and fine-tuning below roughly 2k target tokens, and the gap should narrow as the budget grows. The second is external validity. On historical German, the offsets flagged for verb tags should skew to the right and correlate with measured head–dependent distances. On Early Modern English, prepositional and pronoun offsets should dominate, matching the constructions that normalization misses. If the maps are unstable across bootstrap resamples, or show no relation to syntactic measurements, we will conclude that offset-resolved drift is not identifiable at these data sizes. That would be a negative result worth reporting for anyone designing adaptive windows.