Neural models that inflect words, turning a dictionary form plus grammatical tags into a surface word (Finnish talo + plural genitive → talojen), look nearly solved on standard benchmarks. That picture falls apart when the test words come from roots never seen in training. (Un)solving Morphological Inflection re-split SIGMORPHON 2020 so that no lemma (dictionary form) is shared between train and test. Macro accuracy over 90 languages fell from 0.88 to 0.58, and edit distance rose from 0.27 to 1.18. We propose that a large share of this collapse is error propagation rather than missing knowledge. The model generates a word one character at a time from left to right. When it has never seen a stem, it slips somewhere inside it, and every later character is then predicted on top of that slip. The standard way to repair a cascade of this kind is to add a second-stage corrector. In practice that fails here, because any corrector trained on the first model's outputs on its own training data sees almost no errors: the first model has memorized those lemmas.
The idea is to train a non-causal reviser, a model that reads the complete candidate word together with the lemma and tags and rewrites it, on first-pass outputs produced by models that never saw those lemmas. Concretely, we split the training lemmas into folds and train a first-pass model on all folds but one. Its outputs on the held-out fold then carry exactly the kind of errors that appear at test time on unseen roots. This borrows the held-out stacking trick that Stacked Hierarchical Labeling used to let downstream image classifiers learn to distrust corrupted upstream context. Here we apply it with the held-out folds defined by lemma, and the corrector is bidirectional, so a correct suffix can be used to repair a broken stem.
Core Claim
We make two claims, and both can be falsified. The first is diagnostic. Under lemma splits, the probability that a left-to-right inflection model gets the rest of the word right drops sharply once it has made its first error, well beyond what independent character errors would predict. When the model is fed the correct stem, its affix accuracy mostly recovers. If this holds, the lemma-split gap is substantially a cascade. If it fails, and affix accuracy stays low even given the correct stem, the gap is a failure to compose features, and no revision scheme should help.
The second claim is about the method. A bidirectional reviser trained on lemma-held-out outputs recovers a meaningful part of the lemma-split gap. It should also beat three controls: the same reviser trained on in-sample outputs, the same reviser trained on synthetic random corruptions, and an equal-compute ensemble of first-pass models. We expect the gain to be concentrated on unseen lemmas and long words. We expect it to be small on unseen feature combinations, and we say so in advance.
Relation to Prior Work
Three papers motivate the gap. The lemma-split result above shows it at scale. Can a Transformer Pass the Wug Test? shows that character Transformers lack a bias toward copying the stem unchanged: Finnish accuracy on unseen lemmas is 5.7%. Adding synthetic substring training items raises it to 68.5%, but the authors report that irregular and discontinuous stems remain unsolved. Frequency matters finds that Transformers over-regularize irregular Spanish verbs when data is sparse. Our reviser targets the same failure as hallucination but by a different route. Hallucination changes the training data of the generator. We change what a second model learns to fix, and the two are complementary rather than exclusive.
The closest method is A Framework for Bidirectional Decoding. It generates characters from both word edges inward and marginalizes over generation orders with an $O(|y|^2)$ dynamic program, at $O(|y|^4)$ training cost. It reports gains that widen on words longer than 16 characters, which is direct evidence that decoding order matters for cascades. That work still commits to characters in a single pass, and it was evaluated only on the standard 2022 and 2023 shared-task splits, never on zero-lemma-overlap or unseen-feature splits. Our proposal uses a plain causal generator plus a separate non-causal reviser. Our test is whether a lemma-held-out error distribution, not decoding order, is what fixes unseen-lemma failures. The outside-in decoder is also a first-pass generator we will stack on.
Two papers constrain the design. Context Sensitive Lemmatization Using Two Successive Bidirectional Gated Recurrent Networks predicts a discrete edit tree with bidirectional GRUs and reaches 92.25% on unseen Spanish words. This suggests that classifying edits, rather than generating strings, generalizes to new words, and so our reviser predicts character-level edit operations rather than regenerating from scratch. Morphological Inflection: A Reality Check finds Arabic accuracy falling from about 77% on seen to about 41% on novel feature combinations. Evaluating Morphological Compositional Generalization in Large Language Models finds that morphologically aligned tokens do not rescue GPT-4 on long Turkish and Finnish affix chains. Together these warn that part of the gap lies in feature composition, which revising the stem cannot touch. We therefore measure the two gaps separately instead of claiming both.
On the hierarchical side, A Neural Layered Model for Nested Named Entity Recognition shows the canonical failure of strictly bottom-up stacking: an inner entity missed at a lower layer is never recovered at higher layers. Our reviser is not a further layer in such a stack. It sees the full input again, not just the upstream decision, so it can overrule the first pass.
Research Program
The work has three parts.
The first part is an error-attribution study. We take standard character Transformers and the outside-in decoder, trained on lemma-split and form-split versions of the same data. We align each output to the gold form and to the lemma so that every character is labeled as stem, affix, or boundary. We then measure two quantities. The first is the conditional accuracy of the remaining characters given that the first error occurred at position $t$, compared against an independence baseline. The second is affix accuracy under forced decoding of the gold stem. The difference between free and stem-forced accuracy is our operational measure of the "cascade share" of the gap, reported per language and per split.
The second part is the reviser. We split the training lemmas into $K$ disjoint folds and train first-pass models $g_{-k}$ on the other folds. We collect their beam outputs on fold $k$, so that every training triple (lemma, tags, candidate) has a candidate from a model blind to that lemma. The reviser $r$ is a non-causal encoder over the concatenated lemma, tags, and candidate. It predicts an edit operation per candidate character: keep, delete, substitute, or insert. It is trained to map the candidate to the gold form:
$$\min_r \; \sum_{k=1}^{K} \sum_{(x,y)\in D_k} \mathcal{L}\big(r(x, g_{-k}(x)),\, y\big),$$
where $x$ is the lemma with its tags, $y$ the gold form, $D_k$ fold $k$ of the training data, and $\mathcal{L}$ the edit-sequence cross-entropy. At test time, a first pass trained on all training data produces candidates and $r$ revises them. We will also try one extra round of cross-fitting on revised outputs to see whether iterating helps. Because $K$ first-pass models are trained, the proposal costs roughly $K$ times the training compute of a single generator, and every baseline is given that budget.
The third part is an analysis of where revision helps. We break results down by word length, by the count of affixes (the Turkish and Finnish affix-depth data of Ismayilzada et al.), and by concatenative versus non-concatenative morphology, and we inspect what edits the reviser actually makes.
Baselines
The naive reference point is a single character-level Transformer trained left to right on each split with no augmentation. On the SIGMORPHON 2020 lemma split this class of model yields the reported 0.58 macro accuracy ((Un)solving Morphological Inflection). Liu and Hulden report 5.7% on Finnish and 16.2% on Czech for unseen lemmata. We retrain it ourselves on identical splits so that the numbers are directly comparable.
The strongest data-side competitor is substring hallucination (+hall-2k-substr) from Can a Transformer Pass the Wug Test?. It reports 68.5% Finnish, 54.2% Czech, and 100% Ga on zero-lemma-overlap splits. It is the strongest competitor because it attacks exactly the unseen-stem failure we target, at almost no cost. We run it with the authors' 2k-item recipe on the same CoNLL-SIGMORPHON 2018 and Niger-Congo splits, with the same hyperparameter search budget as our generator.
The strongest decoding-side competitor is the outside-in decoder, xH-Rerank variant, from A Framework for Bidirectional Decoding. It reports gains of 4.7 points over prior systems on the 2022 task and 2.7 points on the 2023 task. No reported result is found on Lacuna for it on zero-lemma-overlap re-splits, so we produce that number ourselves. It is the strongest competitor because it is the one method already designed to stop characters from snowballing.
The compute-matched control is an ensemble of $K$ first-pass generators with majority vote or likelihood reranking. It uses the same $K$ trained models that cross-fitting requires, which isolates whether the gain comes from simply having more models.
Two ablations isolate the key ingredient. The first trains the same reviser on in-sample first-pass outputs; we expect it to learn to copy its input. The second trains it on synthetic corruptions (random character insertions, deletions, and substitutions matched in rate to the observed errors).
The proposal "beats" a baseline when it improves exact-match accuracy on the lemma split by more than seed-to-seed variation across three seeds. It should also beat hallucination alone when stacked on top of it. For the claim to be credible, the gain over the corruption-trained reviser must be positive. Otherwise the benefit comes from generic denoising, not from matching the test-time error distribution.
Why not just…?
Why not just use substring hallucination? This is the most serious threat. On regular concatenative languages it may close most of the gap alone; its 100% on Ga leaves no room. Its authors note it fails on irregular and discontinuous stems, and Kakolu Ramarao et al. show over-regularization persists under sparse data. We therefore run hallucination alone, the reviser alone, and both together, and report per-language. If the reviser adds nothing on top of hallucination for non-concatenative languages, the method claim is refuted.
Why not just use the outside-in bidirectional decoder? Its advantage on long words shows it reduces horizontal cascades, but it still commits to each character once, and nothing in its training exposes it to unseen-lemma errors. The attribution study will show whether its errors on lemma splits still show the conditional-collapse signature. The stacked comparison will show whether revision helps it too. If outside-in decoding alone matches our reviser on lemma splits, we will conclude that decoding order, not error exposure, is the effective ingredient.
Why not just use a large language model? Evaluating Morphological Compositional Generalization in Large Language Models reports GPT-4 lagging humans by about 43% in Turkish and 51% in Finnish on generation, with about 51% coherence on nonce roots versus about 98% for humans. (Un)solving Morphological Inflection found that more training data did not close the lemma-split gap. Scale is therefore not an obvious fix, though we include a prompted LLM as a reference on the Turkish and Finnish affix-depth data.
Why not just train a denoiser on random corruptions? It is cheaper and needs no cross-fitting. Errors on unseen lemmas, however, are structured: an edit distance of 1.18 points to systematic stem and boundary failures, not uniform noise. The corruption-trained ablation tests this directly.
Proposed Experiments
We use four data sources:
- the SIGMORPHON 2020 data under the form versus lemma re-split;
- the CoNLL-SIGMORPHON 2018 six-language and six Niger-Congo zero-overlap splits;
- the SIGMORPHON–UniMorph 2022 Shared Task 0 (paper), whose four conditions (both overlap, lemma overlap, feature overlap, neither) let us separate unseen-lemma from unseen-feature effects;
- the 2023 lemma-split task (paper).
Metrics are exact match and Levenshtein distance, plus the cascade share defined above.
Three checks settle the question. The first is whether the attribution study finds a positive cascade share under lemma splits, one that exceeds the cascade share under form splits. If it does not, we stop claiming that revision should work. The second is the 2022 four-way breakdown. The reviser should gain most in the lemma-unseen, feature-seen condition and least in the feature-unseen condition; the reverse pattern would falsify our account of the mechanism. The third is the ablation ordering. The cross-fitted reviser should beat the corruption-trained reviser, which should beat the in-sample reviser, with all of them above the compute-matched ensemble on unseen lemmas. That ordering is what would show that matching the test-time error distribution, not extra capacity, is doing the work.
References
- (Un)solving Morphological Inflection: Lemma Overlap Artificially Inflates Models' Performance
- Stacked Hierarchical Labeling
- Can a Transformer Pass the Wug Test? Tuning Copying Bias in Neural Morphological Inflection Models
- Frequency matters: Modeling irregular morphological patterns in Spanish with Transformers
- A Framework for Bidirectional Decoding: Case Study in Morphological Inflection
- Context Sensitive Lemmatization Using Two Successive Bidirectional Gated Recurrent Networks
- Morphological Inflection: A Reality Check
- Evaluating Morphological Compositional Generalization in Large Language Models
- A Neural Layered Model for Nested Named Entity Recognition
- SIGMORPHON–UniMorph 2022 Shared Task 0: Generalization and Typologically Diverse Morphological Inflection
- SIGMORPHON-UniMorph 2023 Shared Task 0: Typologically Diverse Morphological Inflection