Suppose someone working with an under-documented language has one day of a native speaker's time. They can spend it labeling ten sentences for a task, or writing down a few hundred unlabeled sentences. Current multilingual NLP gives them no principled way to choose. This proposal measures an exchange rate: how many labeled target-language examples a given amount of unlabeled target text is worth, once a pretrained multilingual encoder has been briefly adapted to that text with a self-supervised objective (a training signal the text provides on its own, such as filling in masked words). We then test whether that rate can be predicted before any data is collected, from cheap properties of the language. The main candidate is how badly the model's subword tokenizer fragments the language's words.
The opportunity comes from two findings that have never been put together. On languages absent from pretraining, a small encoder fine-tuned on a handful of labels beats in-context prompting of larger generative models, and fine-tuning overtakes zero-shot transfer after about 15 labeled examples. That study never let the model see unlabeled target text before fine-tuning. Separately, in dependency parsing, self-supervised objectives on as few as 150 unlabeled target sentences give large gains over zero-shot transfer, but only for languages the model had already seen. Label-efficient learning in other fields points the same way. Brain-signal, CT, and ultrasound foundation models reach near-full-supervision accuracy with 10–20% of labels because pretraining on unlabeled in-domain data comes first. We ask whether the same holds for the languages where multilingual models fail most, and whether one can know in advance when it will pay off.
Core Claim
Our claim is that, for languages unseen or barely seen in pretraining, brief self-supervised adaptation on a few hundred raw target sentences is worth a substantial and predictable number of labeled examples. We expect this exchange rate to be largest where the pretrained tokenizer fragments target words most heavily, and larger for structured token-level tasks (parsing) than for sentence-level classification. If this is right, a practitioner can compute tokenizer fragmentation on a small sample of text and decide whether to spend effort on raw text or on labels. If the rate turns out to be near zero or unpredictable, that is also useful. It would confirm that the "just annotate ten sentences" strategy dominates, and it would do so on languages where that has not yet been shown.
To make the quantity precise, let $F(K, U)$ be test performance of a model fine-tuned on $K$ labeled target examples after adaptation on $U$ unlabeled target sentences. The exchange rate at budget $(K, U)$ is
$$ \rho(K, U) = K' - K, \quad \text{where } F(K', 0) = F(K, U), $$
that is, the number of extra labels the unadapted model would need to match the adapted one. $K'$ is found by interpolating the unadapted learning curve. $\rho$ is capped at the largest labeled budget available, so we report censoring when it occurs.
Relation to Prior Work
Two papers motivate the proposal. Cross-lingual Few-Shot Learning on Unseen Languages compares fine-tuning, prompt tuning, and in-context learning across XLM-R, XGLM, and mT5 on eight NusaX languages absent from pretraining. It finds the 15-sample pivot and shows that transfer is bound to subword overlap. We keep its setting, and add the step it leaves out, unlabeled target adaptation, measuring how much that step moves the curve. Cross-Lingual Universal Dependency Parsing Only From One Monolingual Treebank trains a parser jointly on a source treebank and on masked language modeling plus word ordering (restoring shuffled target sentences) over raw target text, then self-trains. It reaches 80.32 average UAS on 22 UD languages. We borrow its two objectives as our adaptation methods. Where it studies zero-shot parsing on mostly seen languages with Wikipedia-scale text, we study the few-shot regime, languages outside pretraining, and budgets of tens to hundreds of sentences.
Three papers constrain the design. Working Hard or Hardly Working shows that injecting WALS typology into parsers adds little beyond coarse clustering. Ten annotated target sentences gave a 2.3x larger UAS gain than the best typological feature. So typological vectors cannot be the mechanism, and ten labels is the bar any unlabeled-data method must clear. On Language Spaces, Scales and Cross-Lingual Transfer of UD Parsers finds that a structural attachment probe predicts zero-shot parsing transfer best (Spearman −0.66). It also finds that URIEL distances fail within language clusters, where surface statistics such as mean word length and subword productivity do better. This tells us which predictors to test. We predict the value of adaptation, not zero-shot transfer, which that paper leaves open. The Effects of Corpus Choice and Morphosyntax on Multilingual Space Induction shows that cross-lingual space induction is governed by type-token ratio more than language family, and that subword units become overloaded across unrelated contexts. This is the mechanistic reason we expect fragmentation to predict how much raw target text helps: the adaptation step can re-ground overloaded subwords.
Is Prompt-Based Finetuning Always Better than Vanilla Finetuning? supplies a secondary method and a caution. Prompt-style fine-tuning wins at one shot, but its gains shrink for languages far from English. We include it as a fine-tuning variant rather than as the core idea. The grammar-book line, MTOB and code-augmented grammar books, addresses a harder regime with no raw text at all. We treat it as a boundary of our scope rather than a competitor.
Research Program
The experimental grid crosses labeled budget $K \in \{0, 1, 5, 10, 15, 25, 50, 100\}$ with unlabeled budget $U \in \{0, 50, 150, 500\}$, three seeds each, on two task families. For sentiment we use the eight unseen NusaX languages (NusaX). Unlabeled sentences come from the 500-sentence training split with labels stripped, the labeled $K$ are a subset of these, and the 400-sentence test split is never touched. For parsing we use low-resource UD v2.9 target treebanks under the 50,000-token protocol of Samardzic et al. There, unlabeled text is target treebank training sentences with trees removed, and labeled examples are gold trees. English is the source treebank for cross-lingual initialization in both families. The backbone is XLM-R, the strongest model in the unseen-language study, with a mBERT replication to check that conclusions are not backbone-specific.
We compare three adaptation objectives: masked language modeling alone, masked language modeling plus word ordering, and masked language modeling after extending the vocabulary with target-frequent word pieces. The third directly tests the fragmentation hypothesis. If fragmentation drives the effect, vocabulary extension should raise $\rho$ most for the most-fragmented languages.
For prediction, we compute language-level features before adaptation. These are subword fertility (XLM-R tokens per whitespace word), type-token ratio, mean word length, subword productivity, URIEL syntactic distance to English, and the attachment probe score where a treebank exists. We then predict $\rho(10, 150)$ with leave-one-language-out regression. Finally, a diagnostic ties back to the verbalizer failure: after adaptation, we re-run prompt-style fine-tuning with target-language label words. The unseen-language study attributed their failure to ungrounded embeddings, and adaptation should fix exactly that.
Baselines
The naive reference is zero-shot cross-lingual transfer: XLM-R fine-tuned on English NusaX sentiment or English UD, applied to the target with $K = 0, U = 0$. The main baseline is plain few-shot target fine-tuning without adaptation, the curve $F(K, 0)$, which also serves as the measuring stick for $\rho$. On NusaX it is the configuration behind the reported 64.92 average macro-F1 for XLM-R fine-tuning on unseen languages in Cross-lingual Few-Shot Learning on Unseen Languages. Our reproduction must land near that figure before any comparison counts. Full in-language supervision, reported at roughly 80% macro-F1 by NusaX, is the ceiling.
For parsing, the strongest reported system is the multi-task self-training parser at 80.32 average UAS on 22 UD v2.2 languages (One Monolingual Treebank). Its setting differs from ours, and no reported result was found on Lacuna for it on UD v2.9 low-resource targets under a 50,000-token budget. We will therefore rerun it on our targets with its own objectives. SubDP (38.9 Hindi LAS zero-shot) is included only where bitext exists, since it needs parallel text.
For in-context learning, the reported reference is roughly 47–50 average F1 for XGLM and mT5 on unseen NusaX languages. We cite it rather than rerun it at scale. Prompt-style fine-tuning (PROFIT) is run at every $K$.
For fairness, every condition gets the same labeled examples per seed, the same hyperparameter search (a fixed grid of learning rates and epochs selected on the 100-sentence validation split), and the same fine-tuning steps. Adaptation compute is reported separately so the comparison of labels against text is not confounded with extra training. We add a control: masked language modeling on an equal number of English sentences. This separates "any extra training helps" from "target text helps."
Adaptation counts as beating the baseline if $\rho(10, 150) \geq 10$ with confidence intervals excluding zero across languages. In words, 150 raw sentences must be worth at least doubling a ten-label budget. The predictor counts as useful if its held-out rank correlation with $\rho$ exceeds that of URIEL distance.
Why not just…?
Why not just annotate ten more sentences? This is the strongest threat, and the proposal is built around it. Ten target labels beat typological feature injection by 2.3x (Working Hard or Hardly Working), and fine-tuning dominates past 15 examples on unseen languages. Our exchange rate is expressed in exactly those units. If $\rho$ is small, we report that annotation wins. The claim is only that for some identifiable languages raw text is cheaper per unit of accuracy, and that one can tell in advance which languages those are.
Why not just run plain continued masked language modeling and skip the rest? It may well be all that is needed, which is why it is a condition, not a strawman. Word ordering and vocabulary extension earn their place only if they raise $\rho$ beyond masked language modeling, and specifically on the languages the hypotheses name. Word ordering should help on targets whose word order differs from English. Vocabulary extension should help on high-fertility targets. If neither separates from plain masked language modeling, the paper's recommendation becomes the simplest one.
Why not just decide using URIEL or WALS distance? Prior evidence says static databases are noisy within language clusters and mislabel real corpus word orders (On Language Spaces, Scales and Cross-Lingual Transfer of UD Parsers; Working Hard or Hardly Working). Still, URIEL is included as a predictor. Held-out correlation settles whether it matches fragmentation statistics.
Why not just prompt a large language model with the examples or a grammar? On unseen NusaX languages, in-context learning trails small-encoder fine-tuning by about 15 F1. On Kalamang, full-grammar prompting (44.7/45.8 chrF) needs 50k–100k tokens per query and still trails a human learner (MTOB). For languages with some raw text, a permanently adapted small encoder is the cheaper tool. For languages with only a grammar book, our method does not apply, and we say so.
Decisive Checks
Four results decide the proposal. First, on NusaX, does adaptation on 150–500 unlabeled sentences shift the 15-sample pivot earlier and raise the $K = 10$ score above plain fine-tuning, beyond the English-text control? Second, is $\rho$ larger for parsing than for sentiment at matched budgets, as expected if unlabeled text chiefly teaches token-level structure? Third, does subword fertility, or a small set of surface statistics, predict held-out-language $\rho$ better than URIEL, and does vocabulary extension's gain grow with fertility? Fourth, do target-language label words stop hurting prompt-style fine-tuning after adaptation? That would confirm that the earlier failure came from ungrounded embeddings. A negative answer on the first check, with flat learning curves across $U$, would be a clear and publishable finding: for unseen languages, a native speaker's hour is better spent labeling.
References
- Cross-lingual Few-Shot Learning on Unseen Languages
- Cross-lingual Universal Dependency Parsing Only from One Monolingual Treebank
- Working Hard or Hardly Working: Challenges of Integrating Typology into Neural Dependency Parsers
- On Language Spaces, Scales and Cross-Lingual Transfer of UD Parsers
- The Effects of Corpus Choice and Morphosyntax on Multilingual Space Induction
- Is Prompt-Based Finetuning Always Better than Vanilla Finetuning? Insights from Cross-Lingual Language Understanding
- A Benchmark for Learning to Translate a New Language from One Grammar Book
- Read it in Two Steps: Translating Extremely Low-Resource Languages with Code-Augmented Grammar Books
- NusaX: Multilingual Parallel Sentiment Dataset for 10 Indonesian Local Languages
- Substructure Distribution Projection for Zero-Shot Cross-Lingual Dependency Parsing