Grammar Patches: Hand-Written Rules as Cheap Supervision for Parsing Drifted Text

Modern dependency parsers analyze grammar by drawing a labeled arrow from each word to the word it depends on. On text that differs from their training data, they fail mostly on structure, not vocabulary. On late 18th-century German aphorisms, a modern pipeline still lemmatizes at 97.6% accuracy, yet its labeled attachment score (LAS, the fraction of words given the correct head and relation label) falls from 84.6% to 65.4% (Nissim and Salomoni). The words are recognized; the clause structure is not. Linguists often already know what changed: verbs sit at the right edge of nested subordinate clauses, Middle High German has more overt subjects and its own partitive constructions (Dipper et al.), and African-American English on Twitter drops copulas and uses habitual be (Blodgett et al.). The usual way to give a parser that knowledge is to annotate hundreds of sentences in full. That is expensive for historical text, where annotators must be specialists.

We propose that a linguist instead write a few dozen small, local grammar rules ("patches") describing the documented shifts. A trained neural parser would then use these rules as soft bonuses and penalties when it builds the tree. The rules come in at decoding time with tunable weights. A rule that does not apply, or applies wrongly, can therefore only shift scores; it can never block an analysis. This avoids the coverage collapse of full hand-built grammars. The question we care about is economic as much as technical: per hour of expert time, do rule patches buy more parsing accuracy on drifted text than annotated sentences do, and do the two add up?

Core Claim

Our claim is that a small set of hand-written, drift-specific structural rules, applied as weighted soft constraints inside a neural parser's tree search, recovers a substantial share of the accuracy lost under temporal and dialectal shift. At low expert-time budgets it should beat both annotating sentences and using the same rules to rewrite training data. Throughout, the parser is never forced to fail when no rule applies.

This differs from the closest prior work in three ways. The hybrid parsers that motivate us, Zhang and Wang and Øvrelid et al., run a complete broad-coverage grammar (the English Resource Grammar, an XLE grammar written in Lexical Functional Grammar) and feed its output into a statistical parser as features. Their gains are modest on synchronic text: 86.26 to 86.58 UAS on Brown, 85.97 to 87.78 LAS on Tiger. They also inherit roughly 75% coverage out of domain and throughput of about three sentences per second. No such grammar exists for Middle High German or Romantic-era fragments, and building one is out of the question. We need only partial, era-specific rules.

Blodgett et al. use hand-written rules for a different purpose: they rewrite source treebanks so they look like the target dialect, then retrain. We use rules at inference time, and we test this directly against their approach with the same rules.

Dong et al. mask a neural decoder with a pushdown automaton (a parser for a context-free grammar) and reach 100% grammatical output for code. That works because programming languages have an exact grammar known in advance. Natural language under drift does not, which is why our constraints are soft and weighted rather than hard masks.

Research Program

The mechanism is deliberately simple. A biaffine dependency parser trained on modern data assigns a score $\theta(h,d,\ell)$ to every possible arc from head word $h$ to dependent $d$ with label $\ell$. A rule $r$ is a small predicate $\phi_r(x,y)$ over the sentence $x$ and a candidate tree $y$. For example, "inside a clause opened by a subordinating conjunction, the rightmost finite verb heads the clause"; or "a nominal followed directly by an adjective or locative with no verb, in an AAE-tagged tweet, is a copula-less predicate." The decoder picks

$$\hat y = \arg\max_{y} \sum_{(h,d,\ell)\in y} \theta(h,d,\ell) + \sum_{r} \lambda_r \, \phi_r(x,y),$$

where $\lambda_r$ is a per-rule weight. A positive weight rewards trees that satisfy the rule; a negative weight penalizes trees that violate a prohibition. Rules that mention only single arcs fold directly into the arc scores, so exact maximum-spanning-tree decoding still works. Rules over siblings or whole clauses are applied by reranking the parser's k-best trees. Weights are tuned on a development set of at most 50 sentences, using one scalar per rule. Separately, we report a fixed-weight variant with no tuning, so that the gain cannot be credited to the development labels.

Rules are written under a strict protocol. The rule writer works from published descriptions of the shift: Dipper et al.'s documented distributional differences for Middle High German, Nissim and Salomoni's characterization of long, right-nested dependencies, Blodgett et al.'s list of AAE constructions, and standard Latin grammar references. The writer may also inspect the parser's errors on the development sentences only, and logs time spent. Test data is never seen.

Each rule is tagged as one of two kinds. A linguistic-drift rule describes a real grammatical change. An annotation-convention rule records a guideline difference between treebanks, such as Dipper et al.'s obl:arg and nmod:part extensions. This split lets us ablate the two kinds separately. It speaks to the problem Dredze et al. identified, where cross-domain "degradation" was largely conflicting annotation guidelines that no statistical adaptation could repair. If most of our gain comes from convention rules, we will say so: that is a finding about evaluation, not about grammar.

We will build and release three rule sets. The German set covers the chain from modern German to the 18th-century fragments to Middle High German. The Latin set covers the eras in EvaLatin 2024. The AAE set covers Twitter. We also release decoding code that works with any arc-factored parser.

Baselines

The naive reference point is the modern parser applied unchanged. For the 18th-century fragments, the reported figure is 65.4 LAS from modern statistical parsers (Nissim and Salomoni). We rerun our own modern-trained biaffine parser on the same splits rather than relying on that number.

The strongest competitor is fine-tuning on a small number of annotated target sentences. Blodgett et al. report 62.2 LAS on AAE tweets from training on just 250 in-domain tweets. That beat their best cross-domain adaptation at 59.9 LAS, which combined the ARK Twitter tagger, in-domain embeddings, and synthetic data. To make the comparison fair, we match on logged expert time rather than on data size. We measure how many sentences per hour annotators complete on each corpus. We then compare rules written in $t$ hours against sentences annotated in $t$ hours, for $t$ from 1 to 8, using the same base parser and the same development set for tuning. Beating this baseline means a higher LAS at equal hours, with a paired bootstrap confidence interval that excludes zero, for at least the low-budget end of the curve.

The second strong competitor isolates our key ingredient, which is using rules at decoding time. Here we apply the identical rule set the way Blodgett et al. do. The rules rewrite modern treebank sentences into synthetic target-like trees, and the parser is retrained on them. The rule budget is the same and the retraining gets the same compute. If decoding-time use does not beat training-time synthesis with the same rules, then our contribution reduces to "write the rules," and we will report it that way.

On Latin, the relevant prior method is temporal conditioning. Behr concatenated era-aware sentence embeddings into Diaparser and obtained 68.33 LAS on Late Latin versus 68.28 without them, a null result. We reproduce his base model as our Latin starting point. Beating it means a clear LAS gain on the late and medieval subsets.

For Middle High German, we found no reported parsing result on Lacuna for the ReM treebank, so our modern-trained parser is the reference.

Three further controls round out the comparison. The first adds a spelling-normalization front end in the style of VARD (Eisenstein et al.) before parsing. The second is continued pretraining of the encoder on raw historical text. The third combines rules with annotated sentences, to test whether the two are additive. Every system shares one encoder, one decoder, and one tuning budget.

Why not just…?

Why not just annotate a few hundred sentences? This is the real threat. Blodgett et al.'s 250-tweet result shows that small amounts of target supervision beat elaborate adaptation. We do not argue around it; the cost-matched curve is the central experiment. We expect rules to win at one to three hours of expert time, because a single rule fixes every instance of a construction. We expect annotation to catch up as hours grow, and the combination to dominate both. If annotation wins at every budget, the proposal's claim is false, and the rule sets become a diagnostic tool rather than a method.

Why not just use the rules to generate synthetic training data? This is exactly Blodgett et al.'s design, and it is a natural use of the same expert effort. We have two reasons to expect decoding-time use to do better. First, a synthetic rewrite must turn a whole modern tree into a plausible historical one, and that conversion introduces artifacts. Both Zhang and Wang and Øvrelid et al. report that converting between structural representations is error-prone. Second, a soft constraint acts only on target sentences where it fires. The matched-rule ablation in Baselines settles this question directly.

Why not just normalize spelling and use a stronger pretrained encoder? The surface evidence argues against this being enough. Lemmatization is already near 98% on the fragments while parsing loses 19 points (Nissim and Salomoni). Cui et al. find that shifts in grammar rules predict parser degradation better than lexical shift does. Even pretraining on in-domain text did not stop temporal decay for NER on Twitter (Chen et al.). Still, a stronger encoder could absorb some structural drift. That is why continued pretraining and normalization are run as baselines, and why we test rules on top of them rather than only on a weak parser.

Why not just condition on the era? Behr's flat Latin result suggests that a continuous era signal does not tell the parser which arcs to build. Our rules are explicit statements about arcs, which is what that experiment lacked.

Proposed Experiments

We run three settings. The first is the German chain: train on modern German UD; evaluate on the 18th-century fragment treebank (7,337 tokens, using cross-validation because it is tiny) and on the Middle High German ReM treebank (about 29,000 tokens). The second is Latin: EvaLatin 2024, using the classical treebanks for training and the late and medieval treebanks as targets. The third is AAE: the 500-tweet Twitter UD set, with the 250/250 AAE/MAE split. Mainstream American English tweets serve as a check that rules do not harm unshifted text.

Metrics are LAS, UAS, and content-word LAS. We also report per-relation scores and error rate grouped by dependency length, because long arcs are where the historical gap concentrates. Two further statistics are reported for each rule: how often it fires, and its harm rate, the fraction of firings that turn a correct arc into a wrong one.

The decisive checks are four.

The first is the cost-matched curve against annotated sentences, described in Baselines.

The second compares the same rules at decoding time versus as synthetic training data.

The third is a transfer test: rules written only for the 18th-century fragments, applied unchanged to Middle High German. This asks whether patches capture general properties of older German or merely overfit one author's style.

The fourth ablates drift rules against annotation-convention rules, to estimate how much of the measured "temporal drift" is really guideline mismatch.

A clean positive result would show three things: rules beat equal-hour annotation at low budgets, decoding-time use beats synthesis, and the harm rate stays well below the rate at which rules make correct fixes. A clean negative result would also be informative: it would tell practitioners in historical NLP to spend their expert hours annotating sentences rather than writing grammar.