Round-Trip Pointer Models for Cleaning and Completing Unsupervised Paradigms

Many of the world's languages have words that change form for case, tense, person, and number. A single verb may have dozens of forms. Unsupervised paradigm completion asks a system to fill in a word's full table of forms (its paradigm) using only raw text and a list of dictionary forms (lemmas), with no labeled examples. The standard method is a four-stage pipeline from Unsupervised Morphological Paradigm Completion. It mines string edits between similar words, guesses which words are forms of which lemma, groups the edits into grammatical slots, and then trains a character-level generator on the resulting guessed (lemma, slot, form) triples. In the SIGMORPHON 2020 shared task, no submitted system beat this pipeline on average (shared task overview). That includes the best one, a pointer-generator: a decoder that, at each output character, mixes generating from an alphabet with copying a character from the input.

The pattern of results suggests the bottleneck is not the generator but the noisy training pairs it is fed. Making the final generator better only moved accuracy by about a point. This note proposes using the pointer-generator in a different role. Instead of a last-stage form producer, it becomes a filter that checks the pipeline's guesses and a completer that copies from every form of a word already attested in text. The filter works by round trips: a model trained to map form A to form B, and also B back to A, should reconstruct genuine pairs in both directions. Spurious pairs, such as derivations like walk → walker or accidental look-alikes, should fail at least one leg.

Core Claim

A character-level pointer-generator trained bidirectionally across all attested cells of each induced paradigm can do two jobs. First, it can serve as a cross-fitted consistency scorer that removes corrupted pseudo-pairs and incoherent slots before final training. Second, it can serve as a multi-source completer that copies stem material from whichever attested forms are most informative, not only from the lemma. The prediction is that the filtering step, and not the generator architecture, is what closes the gap to the official baseline on SIGMORPHON 2020 Task 2. We also predict that multi-source copying helps most where the lemma is a poor predictor of other cells, such as stem-changing or templatic morphology.

Of the three input directions, the pointer-generator is the core mechanism and unsupervised paradigm completion is the application. Bidirectional modeling enters as the paradigmatic direction of prediction: forms predicting each other both ways, plus a standard bidirectional character encoder. Sentence-level bidirectional context is deliberately left out of scope. The reasons are given below.

Relation to Prior Work

The closest work is the pair of 2020 Task 2 systems. IMS–CUBoulder kept the baseline's first three stages and replaced its generator with a character pointer-generator (IMS-CUB2). That system was the best submission and beat its own vanilla attention LSTM by 3.6% macro BMAcc, but it still fell 1.03% short of BL-1 and 0.3% short of BL-2. NYU-CUBoulder put a pointer-generator into a Transformer and added reinflection multitask training. Its best Task 2 entry was nevertheless an ensemble of vanilla Transformers.

These papers supply our method: we reuse the pointer-generator decoder. They also motivate our hypothesis, because both consume upstream pseudo-data passively and both stay near the baseline. Our proposal differs in three ways. Training pairs run between arbitrary attested forms in both directions, not lemma to form only. The model's scores feed back to prune the pseudo-data. And completion copies from several attested sources at once.

The idea that non-lemma forms can predict other cells better than the lemma comes from supervised work. Neural Graphical Models over Strings for Principal Parts Morphological Paradigm Completion models paradigm cells jointly as a graph with sequence-to-sequence edges. Frugal Paradigm Completion exploits the fact that cells predict each other mutually to cut human elicitation by up to 63%. Both rely on gold UniMorph tables or a human oracle. We carry the insight into the setting where the paradigm tables are themselves induced and noisy, and we replace fixed graph edges with a single copy-attention model over a variable set of sources.

Two papers constrain the design. Learning Bidirectional Morphological Inflection like Humans finds that neural inflection models are systematically more accurate in one direction (present → past) than the reverse. A naive round-trip score would therefore penalize genuine pairs simply for lying in the hard direction. We handle this by normalizing scores within each source-slot/target-slot direction, as described in the research program.

Pointer-Generator Networks for Low-Resource Machine Translation: Don't Copy That! and Get To The Point both warn that copy models drift toward verbatim copying. In our setting that means near-identity pairs would score trivially well. We therefore score pairs relative to a copy-only reference, so that identity is not rewarded.

Sentence-context methods are a known alternative for disambiguating slots. Contextualization of Morphological Inflection uses a bidirectional LSTM over the sentence to predict latent tags, and Modeling Syntactic Context Improves Morphological Segmentation uses agreement with neighboring words. The first is supervised with UD tags and reaches only 61.1% on Basque and 59.3% on Finnish even so. We treat it as a later extension rather than a core ingredient, because the Bible corpora in Task 2 are small and the pipeline already uses an unsupervised POS tagger for slot discovery. Finally, A Pointer Network Architecture for Joint Morphological Segmentation and Tagging shows that pointing among valid candidates cuts errors sharply on Turkish, but it needs a hand-built analyzer. Our copy targets are characters of attested words, so no analyzer is required.

Research Program

The pipeline's first three stages are run unchanged on each language's JHU Bible text and lemma list. This yields induced slots and, for each lemma, a partial paradigm of corpus-attested forms assigned to slots.

From every partial paradigm we create training examples for every ordered pair of attested cells. Each example has a source form with its slot tag, a target slot tag, and the target form. The model is a bidirectional character LSTM encoder with a pointer-generator decoder. At completion time it receives a set of $K$ attested source forms of the same lemma. Each source is encoded separately, and decoder attention runs over all source characters jointly. The output distribution is

$$P(c) = p_{gen}\,P_{vocab}(c) + (1-p_{gen})\sum_{k=1}^{K}\sum_{i:\,x^{(k)}_i=c} a^{(k)}_i ,$$

where $c$ is the next output character, $p_{gen}$ is the learned generate-versus-copy gate, $x^{(k)}_i$ is the $i$-th character of source $k$, and $a^{(k)}_i$ is its attention weight, normalized over all $K$ sources together. When $K=1$ and the source is the lemma, this reduces to the IMS-CUB2 setup.

Filtering uses cross-fitting. Lemmas are split into folds, and pairs from each fold are scored by a model trained on the others, so that a pair's score does not reflect memorization of that pair. For a pseudo-pair $(a,b)$ with slots $(s,t)$, the score is

$$r(a,b) = z_{s\to t}\big(\log P(b\mid a) - \log Q(b\mid a)\big) + z_{t\to s}\big(\log P(a\mid b) - \log Q(a\mid b)\big),$$

where $Q$ is a copy-only reference model with $p_{gen}$ fixed at 0. Subtracting $\log Q$ removes credit for trivial copying. Each $z$ standardizes scores within one direction between one slot pair, which absorbs the directional asymmetry noted above.

Pairs below a threshold are dropped, and so are slots in which most pairs fail. Filtering removes pairs from the training data, not words from the corpus. The model is then retrained, and the cycle repeats for at most three rounds. The threshold and round count are tuned only on the five development languages (Maltese, Persian, Portuguese, Russian, Swedish), where UniMorph lets us measure pseudo-pair precision directly.

Completion generates each unattested cell from all attested forms of the lemma. Analysis asks three questions: where the gate copies from, which source receives attention for which target slot, and whether filtering removes the derivational pairs that the 2021 clustering overview identifies as a chronic confusion.

Baselines

The naive reference point is the official pipeline itself, BL-1 and BL-2 of the 2020 shared task. Their generator is trained on unfiltered (lemma, slot, form) triples and inflects from the lemma alone. Because no submission beat them on average, they are also the operational strongest competitor. The research file shows BL-1 at about 38.4% macro BMAcc and IMS-CUB2 at about 37.4% in a summary diagram. The papers themselves report the comparisons in relative terms, so we treat the relative gaps as the authoritative figures. For English, Jin et al. report 74% BMAcc for the pipeline.

The strongest neural competitor is IMS-CUB2 (IMS–CUBoulder), a lemma-only character pointer-generator on the same pseudo-data. Its reported shortfall is 1.03% BMAcc below BL-1 and 0.3% below BL-2. We will reimplement it inside our codebase with the same encoder size, character vocabulary, and tuning budget as our model. Hyperparameters for every system are selected on the dev languages only, with the same number of trials per system, so the comparison isolates data and sources rather than tuning effort.

We also include the vanilla-Transformer ensemble approach of NYU-CUBoulder. To keep it fair, our own model is ensembled to the same size when compared against it.

Three ablations isolate the key ingredients. The first applies round-trip filtering but keeps lemma-only generation, which tests filtering alone. The second keeps multi-source generation but removes filtering. The third uses multi-source generation with the copy term removed ($p_{gen}=1$), which tests whether copying, rather than simply having more sources, does the work.

Beating the competition means two things. Our system must exceed BL-1's macro BMAcc on the nine test languages, which no 2020 system did, with a margin larger than the spread across random seeds. And the filtering-only ablation must account for most of that gain. If only multi-source generation helps, the core claim about upstream noise is wrong and we will say so.

Why not just…?

Why not just train a bigger or ensembled generator on the existing pseudo-data? The 2020 results argue against it. Three different generator designs (IMS-CUB2, the NYU ensembles, and the baseline's own generator) cluster within roughly a point of each other. A stronger copy model also reproduces corrupted stems more faithfully, which is the upstream error propagation the file documents. The no-filter multi-source ablation, run at matched ensemble size, tests this directly.

Why not just filter pseudo-pairs with simple counts, such as dropping edit trees seen fewer than $n$ times or requiring slot frequency thresholds? This is a genuine threat. Frequency filtering is cheap and may remove much of the same noise. We will tune such a filter on the dev languages with the same budget as our threshold and report pseudo-pair precision and recall against UniMorph for both. Our expected advantage lies in pairs that are frequent but wrong, such as productive derivational suffixes like -er, which counts cannot reject but a reverse mapping often can. If the count filter matches round-trip filtering on test BMAcc, the conclusion is that the gain comes from filtering, not from the neural scorer, and the proposal will report it that way.

Why not just apply the baseline's edit trees from attested non-lemma forms, a symbolic version of principal parts? Symbolic rules avoid training altogether. However, transformation-rule systems overgenerate and degrade under iterated application, and they handle internal stem changes poorly. We will include this symbolic multi-source variant as an additional baseline. If it matches our multi-source pointer model, the value of copy attention is limited to the scorer.

Why not just add sentence context with a bidirectional encoder, as in contextual inflection? Vylomova et al. needed gold tags to make context pay off, and they still did poorly on agglutinative languages. In an unsupervised setting with small Bible corpora, the signal is weak, and the pipeline already uses distributional tags for slot discovery. We leave this extension for later rather than claim it.

Proposed Experiments

The main evaluation is SIGMORPHON 2020 Task 2: macro BMAcc over the nine test languages (Basque, Bulgarian, English, Finnish, German, Kannada, Navajo, Spanish, Turkish), plus per-language results, with all tuning done on the five development languages.

The first decisive check is the headline comparison against BL-1, BL-2, and the reimplemented IMS-CUB2 across five seeds.

The second check is diagnostic and runs on the dev languages. We measure the UniMorph precision of pseudo-pairs before and after each filtering round, and the fraction of derivational pairs removed. If filtering does not raise precision, any downstream gain must come from elsewhere.

The third check splits test cells by whether the lemma or some other attested form shares more characters with the gold target. The multi-source model should gain mainly in the second group.

As a secondary test of filtering, the slot-filtered paradigms are converted into clusters and scored with best-match F1 on the SIGMORPHON 2021 clustering task (findings). The reference point there is the Edinburgh system's reported 67.96% F1 and the transformation-rule system's 66.12%.

We expect Basque and Navajo to remain hard because many cells have no attested source at all. We will report them separately rather than let the macro average hide them.