Documenting an endangered language means producing interlinear glossed text: each word is split into morphemes, and each morpheme gets a label. The label is either a grammatical tag such as PST or ERG, or an English meaning for the stem. Expert linguists write these labels by hand, and for many languages only a few dozen sentences ever get annotated. Multilingual models pretrained on hundreds of thousands of glossed sentences help, but they have a specific weakness. The grammatical tags come from a small shared inventory and should carry over between languages. In practice, a model's internal representation of "this morpheme is past tense" is shaped by whichever well-documented language dominated its training data. When a new language arrives with 31 annotated sentences, the tag predictor is still tuned to the grammar of that dominant language.
We propose to fix this with a targeted form of adversarial training. Adversarial domain adaptation normally adds a "domain discriminator," a small classifier that tries to guess which domain an internal representation came from, and trains the main model to fool it. Here the domain is the language. We make two changes to the usual recipe. First, we apply the adversary only at the output positions where the model emits a closed-class grammatical tag, and leave stem positions alone. Second, the discriminator is told the gold tag. The model is therefore pushed to make "past tense in Tsez" and "past tense in Arapaho" look alike, without being pushed to make Tsez and Arapaho look alike overall. The claim is that this tag-conditioned, tag-restricted invariance improves grammatical-tag accuracy on unseen, extremely low-resource languages. Plain adversarial alignment cannot deliver that gain, and it is known to damage fine-grained lexical tasks.
Why this and not the obvious combination
The obvious idea at this intersection is to bolt a gradient-reversal language discriminator onto a multilingual glossing model. The closest prior work suggests it would fail. Adversarial Neural Networks for Cross-lingual Sequence Tagging trains a token-level language discriminator for French↔Spanish transfer. Its gains are on structural tasks, while POS tagging collapses to 45% accuracy. That is the lexical, fine-grained regime that glossing lives in. The same paper also reports that GAN and WGAN adversaries overpower the feature extractor and leave it with "dead gradients," while plain gradient reversal is more stable. We take gradient reversal from it as a methodological constraint.
Representation Alignment and Adversarial Networks for Cross-lingual Dependency Parsing combines an encoder-level adversary with dictionary-based input alignment for Chinese→Vietnamese parsing. The adversary alone adds little, +0.51 LAS relative to the full model's contribution, and the approach relies on a bilingual dictionary that endangered languages lack. Cross-lingual Dependency Parsing as Domain Adaptation frames parser transfer as feature-space alignment. All three align whole representations across languages. None addresses morpheme glossing, and none separates what should be shared from what should not.
We read the POS collapse as a symptom of label shift. When tag frequencies differ across languages, forcing the full distribution of representations to match across languages also erases information about the tags. Conditioning the discriminator on the tag addresses this directly. It asks only that representations be indistinguishable across languages among morphemes carrying the same tag.
The glossing literature supplies the task split and the testbed. Towards Multilingual Interlinear Morphological Glossing and GlossLM: Multilingual Pretraining for Low-Resource Interlinear Glossing both identify the same divide. Grammatical tags form a closed, transferable inventory. Stems are open-ended and best recovered from the free translation. GlossLM also supplies what our training depends on: a 450k-example, 1,800-language corpus with more than 80% of labels normalized to the UniMorph schema. That normalization is what makes "the same tag across languages" meaningful. The corpus is heavily skewed toward a few languages such as Arapaho, which is the imbalance we target. Stem prediction is left to the existing translation-anchoring machinery of Automatic Interlinear Glossing for Under-Resourced Languages Leveraging Translations.
Core Claim
Let $h_t$ be the decoder hidden state at output position $t$. Let $y_t$ be the gold gloss token at that position, and let $\ell$ be the language of the sentence. Let $\mathcal{G}$ be the set of positions whose gold gloss is a UniMorph grammatical tag.
The discriminator $D$ receives the pair $(h_t, y_t)$ and predicts $\ell$. Its loss is the cross-entropy $\mathcal{L}_{D}$ summed over $t \in \mathcal{G}$. The glossing model, with parameters $\theta$, minimizes its usual sequence loss $\mathcal{L}_{gloss}$ while maximizing $\mathcal{L}_D$ through a gradient reversal layer with weight $\lambda$:
$$\min_\theta \; \mathcal{L}_{gloss}(\theta) \;-\; \lambda \sum_{t \in \mathcal{G}} \mathcal{L}_D\big(D(h_t(\theta), y_t), \ell\big).$$
At equilibrium, the distribution of $h_t$ given the tag $y_t$ no longer depends on the language. The distribution of $h_t$ over all positions still can, so stems and tag frequencies may remain language-specific.
The prediction follows from this. On an unseen language fine-tuned on 31–100 sentences, the model should get more of its grammatical tags right, and its stem accuracy should not drop. Neither unconditional adversaries nor better data balancing should reproduce this pattern.
Research Program
The backbone is ByT5, the byte-level encoder–decoder used by GlossLM. It accepts unsegmented text, so the same model runs on both SIGMORPHON tracks. We place the adversary on decoder states rather than encoder states. On unsegmented input, only the decoder has a well-defined position for each gloss.
During multilingual pretraining on the GlossLM corpus, the seven SIGMORPHON 2023 languages are held out entirely. The discriminator is a two-layer MLP. With 1,800 languages, it predicts identity among the languages that have at least a few hundred tag tokens, and pools the rest into an "other" class. As a check, we also run a variant that predicts language family instead.
At training time, gold labels tell us which positions belong to $\mathcal{G}$. At inference no discriminator is used. We warm up $\lambda$ from zero and select it on held-out pretraining languages, never on the test languages.
After pretraining, each held-out language is fine-tuned at 31, 100, and full training size. The adversary stays on during fine-tuning, with the target language added as a class. This follows the few-shot simulations of Findings of the SIGMORPHON 2023 Shared Task on Interlinear Glossing and Multiple Sources are Better Than One: Incorporating External Knowledge in Low-Resource Glossing.
The analysis side has two parts. First, a linear probe measures how much language identity can be read off tag-position states given the tag, before and after training. Second, we break down accuracy by gloss type: grammatical tags versus stems, with stems further split into seen and unseen.
Baselines
The naive reference point is ByT5 fine-tuned directly on each target language with no multilingual pretraining. It shows how much any transfer buys at 31 sentences.
The strongest existing trained method is GlossLM pretraining followed by target fine-tuning (GlossLM). The file reports that it sets a new best on 5 of 7 SIGMORPHON 2023 languages, and exceeds the previous best on unsegmented text by up to 6.6 points of morpheme accuracy. No reported absolute per-language accuracy for GlossLM was found on Lacuna. We will therefore retrain it ourselves, using the same corpus, held-out split, compute, and hyperparameter budget as our model. The only difference will be the adversarial term.
For external reference we report two published results. The shared-task winner TÜ-CL reaches 71.30% micro-averaged word-level accuracy (Findings of the SIGMORPHON 2023 Shared Task). At the extreme low end, the latent-variable CRF of Towards Multilingual Interlinear Morphological Glossing reaches about 52% on Gitksan from 31 sentences, where a Transformer drops to about 25%. Beating the retrained GlossLM baseline means a statistically significant gain in grammatical-tag morpheme accuracy, averaged over the seven languages at 31 and 100 sentences, with seeds and bootstrap intervals, and no significant loss on stems. Matching or beating the Gitksan CRF number at 31 sentences would be a secondary target.
Three ablations isolate the key ingredient. An unconditional adversary applied at all positions reproduces the recipe of Adel et al., and we expect it to hurt stems. An unconditional adversary restricted to tag positions tests whether conditioning on the tag matters. A factored model with separate tag and stem output heads and no adversary tests whether the gain comes from the split rather than from the invariance. All ablations use identical budgets.
Why not just…?
Why not just fine-tune GlossLM and accept its errors? That is our main baseline, and the comparison above settles it. The file gives specific reasons to expect headroom. GlossLM concentrates its failures on low-data languages, the corpus is skewed toward Arapaho, and the shared-task organizers note that pretrained ByT5 underperformed specialized architectures. If our model does not beat the retrained GlossLM on grammatical tags at 31–100 sentences, the idea fails.
Why not just rebalance the pretraining data with temperature or language-balanced sampling? This is the most serious threat. If Arapaho dominance is the whole problem, resampling fixes it at zero complexity. We will run temperature sampling at several temperatures as an extra baseline, and also combine it with our method. The decisive evidence comes in two forms. One is whether language identity remains decodable from tag-position states under resampling. The other is whether the adversary adds accuracy on top of the best resampling setting. If it adds nothing, we will report that resampling suffices.
Why not just give the model a language embedding, or split the output into tag and stem heads? Conditioning on language does the opposite of what we want: it licenses language-specific tag representations, and an unseen language has no trained embedding. The factored-heads model is included as an ablation for exactly this question. If it matches the adversarial model, the contribution is the split, not the adversary, and we will say so.
Why not just prompt an LLM word by word? Prompt and circumstance beats the shared-task baseline everywhere, and its 3-best oracle beats the winner on 5 of 7 languages. LLM post-correction adds about 10 points at 100 sentences (Yang et al.). But both works flag the cost and reproducibility problems of proprietary APIs, and performance drops for Spanish-translated Uspanteko. Our model is a cheap, open first pass that such post-correction could still refine. We will report one LLM post-correction run on top of our model and on top of GlossLM, to check that the gains are complementary rather than redundant.
Decisive Checks
The first check is the probe. Tag-conditioned language identity should fall toward chance under our method, while unconditioned language identity at stem positions should stay high. If both fall, the adversary is erasing more than intended.
The second check is the per-type breakdown. Our gain should appear in grammatical tags. It should be largest for typologically distant held-out languages such as Gitksan and Natügu, and absent or negative under the unconditional adversary.
The third check is scaling with target data. The advantage over GlossLM should shrink from 31 sentences to the full training set. Tag-level transfer should matter least once the target language supplies its own examples, and a gap that fails to shrink would suggest the gain comes from something other than transfer.
We run all conditions on both the segmented and unsegmented tracks. This shows whether the effect survives when the model must also find morpheme boundaries, which the SIGMORPHON findings identify as the real field-documentation bottleneck.
References
- Adversarial Neural Networks for Cross-lingual Sequence Tagging
- Representation Alignment and Adversarial Networks for Cross-lingual Dependency Parsing
- Cross-lingual Universal Dependency Parsing Only from One Monolingual Treebank
- Towards Multilingual Interlinear Morphological Glossing
- GlossLM: A Massively Multilingual Corpus and Pretrained Model for Interlinear Glossed Text
- Automatic Interlinear Glossing for Under-Resourced Languages Leveraging Translations
- Findings of the SIGMORPHON 2023 Shared Task on Interlinear Glossing
- Multiple Sources are Better Than One: Incorporating External Knowledge in Low-Resource Glossing
- Prompt and circumstance: A word-by-word LLM prompting approach to interlinear glossing for low-resource languages