When Should a Multilingual Model Adapt to an Unlabeled Language?

A model fine-tuned for a task in one language and then applied to another language it was never trained on for that task (zero-shot cross-lingual transfer) often does poorly on low-resource languages. On the AfriSenti zero-shot sentiment track, the best system averaged only 58.15 F1 on Oromo and Tigrinya (SemEval-2023 Task 12). One cheap fix from computer vision is test-time entropy minimization: while the model runs on unlabeled target inputs, nudge a few parameters so its predictions become more confident (Tent). The trick has a known way of failing. Theory and experiments show that it improves a model only if the model is already "decently" right on the target, and that it makes a weak model worse (Self-training Avoids Using Spurious Features Under Domain Shift). Low-resource transfer is exactly the regime where starting accuracy is often weak.

This note proposes treating that failure as the main object of study rather than a footnote. We will map, across dozens of real African-language transfer pairs, where test-time entropy minimization helps and where it hurts. We will then build a label-free gate that decides, for each target language, whether to adapt and when to stop. In short, the claim is this: confidence-sharpening is a useful but dangerous tool for unlabeled languages, and the useful part comes from knowing, without labels, when not to use it.

Core Claim

Two predictions follow from the initialization result of Chen et al. First, the gain from test-time entropy minimization will change sign as a function of the unadapted model's target accuracy. Second, that sign can be predicted from unlabeled signals well enough that a gated adapter almost never does worse than doing nothing, while keeping most of the gain on languages where adaptation helps. We also make a sharper claim: the gain is not just a correction of mismatched label frequencies, and not just exposure to target-language text. Both have cheaper explanations, and the experiments test them directly.

The nearest prior work applies SHOT to synthetic English dialects (Harnessing Test-time Adaptation for NLU tasks Involving Dialects of English). SHOT minimizes per-example entropy plus a term that keeps predicted classes diverse, adapting a whole feature block and adding k-means pseudo-labels. That study finds an average gain of 0.98 points. It also finds that gains grow with the size of the dialect gap. It covers only rule-transformed English with a shared vocabulary and script, and it never sees the low-accuracy regime where self-training is predicted to break. The other existing bridge, EVALM, uses label entropy as an offline, supervised criterion for choosing which words to add to the vocabulary. It never optimizes entropy on unlabeled text. One-shot Entropy Minimization for Language Model Reasoning shows that about 10 entropy steps can sharpen knowledge a large language model already has. It also states the limit we expect to hit: entropy minimization cannot create knowledge that is absent, and its results vary up to twofold across random seeds.

Research Program

The adapter itself is deliberately simple, so that the scientific content sits in the gate and in the failure map. We start from AfroXLMR-large fine-tuned on labeled source languages. At test time we update only the LayerNorm scale and shift parameters, which is the transformer counterpart of Tent's batch-norm parameters. We minimize the following loss on unlabeled target batches:

$$\mathcal{L} = \frac{1}{B}\sum_{i=1}^{B} H\big(p_\theta(y \mid x_i)\big) \;-\; \lambda\, H\big(\bar p_\theta\big)$$

Here $p_\theta(y \mid x_i)$ is the predicted class distribution for target input $x_i$, $B$ is the batch size, and $H$ is Shannon entropy. The term $\bar p_\theta$ is the average prediction over the batch, so its entropy rewards using all classes and blocks the trivial solution of predicting one class everywhere. The weight $\lambda$ trades off the two terms. For NER we apply the same loss to each token's predicted label.

The gate combines three unlabeled signals, each computed on the target text alone.

The first is the Weighted Flips accuracy estimate. That paper documents a two-phase pattern: target embeddings first tighten around source class centers, then drift away and collapse. Weighted Flips is built to track this trajectory. The authors note it works less well on small datasets. Many AfriSenti languages are small (Xitsonga has about 1.2k tweets), so whether the signal survives here is a test, not an assumption.

The second signal is agreement among source models fine-tuned from different source languages. Disagreement is a direct label-free proxy for weak initialization.

The third is subword fertility, meaning the number of subword pieces the tokenizer produces per target word. EVALM argues that heavily fragmented words carry noisy, high-entropy evidence, so we expect fertility to predict cases where sharpening amplifies noise.

The gate makes two decisions per target language: whether to adapt at all, and at which step to stop. It is fit on a set of development languages with labels and then frozen. It is evaluated only on held-out target languages whose labels are never used.

To trace the full failure curve, we need to vary starting accuracy on purpose. For each target language we use several source configurations that differ in how good the transfer is. These are English-only, a typologically distant African language, a related language, and all other languages pooled. MasakhaNER already shows that the choice of source matters: transferring from Hausa beat English or Swahili by over 16% (MasakhaNER). This yields many (source, target) points spread along the initial-accuracy axis. That gives the cross-lingual counterpart of the 95% versus 97% spurious-correlation contrast on Colored MNIST, where self-training raised accuracy from 82% to 91% in one case and lowered it from 72% to 67% in the other.

Baselines

The naive reference is unadapted zero-shot transfer from the same fine-tuned checkpoint. Every method below is reported as a change relative to it, on the same languages, with five or more seeds, because one-shot entropy methods are known to be unstable across seeds.

The strongest published zero-shot reference on our main benchmark is the AfriSenti Task C winner, UCAS-IIE-NLP, at 58.15 average F1 on Oromo and Tigrinya (SemEval-2023 Task 12). We will replicate the Task C protocol exactly. Matching or exceeding 58.15 with our backbone plus gated adaptation counts as competitive. Clearly exceeding it counts as beating it, with the caveat that the winning system's other engineering choices may differ from ours. As a supervised ceiling we report AfroXLMR-large trained with in-language labels, which reaches 71.2 macro-F1 across 14 languages (AfriSenti).

The closest method baseline is SHOT, run as in the English-dialect study: it adapts the encoder and bottleneck and uses k-means pseudo-labels. It gets the same unlabeled target text, the same number of gradient steps, and the same tuning budget on development languages.

We also run two cheaper baselines that are serious threats. One is label-prior re-estimation: we re-weight the frozen model's output probabilities so their average matches an estimated target class distribution, with no parameter updates. The other is task-adaptive masked language modeling on exactly the same unlabeled target text, using the same compute, after which the frozen task head is reused.

Hard-threshold pseudo-labeling with matched steps is included as well. It provides an empirical check on the theoretical equivalence between continuous pseudo-labeling and entropy minimization.

Two ablations isolate the gate. Ungated entropy minimization with a fixed 10-step budget shows what the gate adds. An oracle gate, which uses target labels to pick adapt-or-not and the stopping step, bounds what any gate could achieve.

The gated method beats the field if two things both hold. It must improve mean F1 over zero-shot, prior correction, and task-adaptive masked language modeling on held-out languages. And on no held-out language may it fall below zero-shot by more than seed noise. The ungated variant is expected to fail this second condition.

Why not just…?

Why not just run masked-language-model adaptation on the target text? Language-adaptive pretraining is strong when text is plentiful. Examples include the Nguni gains of up to 5.2 F1 from 261M tokens (NGLUEni) and the AfriSenti Task A and B wins built on adaptive pretraining. In the test-time setting, though, only the test stream is available, often about a thousand tweets. This is a real threat, so it runs head-to-head on identical text and compute. If it matches gated entropy minimization, the conclusion is that exposure to the language, not confidence-sharpening, drives the gain. We will report that outcome as such.

Why not just correct the label frequencies? The diversity term in our loss partly acts as a prior correction. Sentiment transfer errors may consist largely of systematic over-prediction of one class, such as neutral. If output re-weighting with no parameter updates recovers most of the gain, the method is unnecessary for classification. We would then shift the claim to NER, where per-token label shift is a less complete explanation. This comparison is the first one we will run.

Why not just pseudo-label with a confidence threshold? Chen et al. prove that continuous pseudo-labeling is equivalent to entropy minimization, so this alternative carries the same initialization risk. Entropy Enigma further reports that entropy minimization still works when the most confident examples are removed from the loss, which weakens the intuition that confident examples are what matter. Thresholding does not address our question. The gate can wrap either method, and the matched-step comparison shows whether the continuous loss helps.

Why not just pick a better source language? Source choice matters a great deal (the Hausa result above). Choosing a good source is complementary to our method, and it is also how we vary initial accuracy across experiments. We will check whether gated adaptation adds anything on top of the best single source, chosen without target labels by the source-agreement signal.

Proposed Experiments

The benchmarks are AfriSenti, using leave-one-language-out across its 14 languages plus the exact Task C split; MasakhaNER, with cross-lingual transfer and the Wikipedia-to-news shift where F1 often falls below 40%; and NGLUEni zero-shot transfer to isiNdebele and Siswati. The primary metric is macro-F1, or entity F1 for NER. Secondary measures are the gate's AUROC for predicting harmful adaptation and its regret relative to the oracle gate.

Three checks decide the outcome. First, plot the gain from ungated adaptation against unadapted target accuracy over all (source, target) pairs. A sign change near a stable accuracy level would confirm the initialization story on real languages. Second, compare the gain against prior correction and task-adaptive masked language modeling. If either explains most of it, the method claim fails, though the failure map remains a contribution. Third, test whether the gate's held-out AUROC stays useful on the smallest languages, where the Weighted Flips signal is expected to be weakest. Fertility and source-agreement must carry the gate there, or the gate is not fit for the languages that most need it.