Cloze Tasks Through Language Adapters: Making the Output Head Speak Unseen Languages

Few-shot classification in languages that a multilingual model never saw in pre-training fails for two separate reasons, and current methods each fix only one. The first reason is capacity. One shared network has to cover a hundred languages, so rare languages end up with diluted representations and get split into meaningless subword fragments. Language adapters address this. They are small per-language modules inserted into a frozen model and trained on unlabeled text. The second reason is an objective mismatch. Standard fine-tuning throws away the model's word-prediction head and trains a new classifier from random weights, which wastes the handful of labels available. Cloze prompting addresses this by rephrasing the task as fill-in-the-blank ("The food was ⟨blank⟩." with answers "great" or "terrible"). This note argues that the two fixes are complementary in one specific way. A language adapter is trained by masked word prediction, which is exactly the objective a cloze task uses. So the adapter should do more than enrich the network's internal representations. It should also make the word-prediction head able to score label words in the target language.

The concrete bet is this. Take the adapter-based transfer recipe of MAD-X, keep the pretrained prediction head instead of replacing it, and pose the task as a cloze. Under that setup, label words written in an unseen language should go from harmful to helpful. That reversal is the falsifiable signature of the idea. Cross-lingual Few-Shot Learning on Unseen Languages found that translating label words into Balinese hurt performance compared with keeping English labels, because the shared head has no grounding for those words. We predict the sign of that effect flips once the target-language adapter, including its output-side component, sits between the model and the head.

Core Claim

Modular language adaptation and cloze-style task alignment combine more than additively when the language module is trained by the same masked-prediction objective that the task head reuses. When MAD-X swaps an English adapter for a target-language adapter, its linear classifier receives representations it was never trained on. A cloze head, by contrast, receives representations that the target adapter was explicitly trained to make predictable through that very head. This matters most where the capacity bottleneck is worst: unseen languages, heavy subword fragmentation, and fewer than about 15 labels. In that regime, Cross-lingual Few-Shot Learning on Unseen Languages reports that no method was convincingly ahead. Below 15 samples, zero-shot transfer and in-context learning were competitive with fine-tuning. Above it, plain fine-tuning won.

The three directions play different roles here. Pre-training task misalignment supplies the mechanism: keep the head and keep the objective. Language-specific modularity supplies the means of grounding that head per language. The capacity bottleneck defines the regime and the diagnostic. We expect the gains to grow with how badly a language is fragmented and starved by the shared vocabulary.

Relation to Prior Work

Two lines of prior work motivate the proposal, and each stops short in a different place.

The modular line works inside the network but discards the prediction head. MAD-X stacks task adapters on masked-prediction-trained language adapters. It adds invertible adapters at the embedding and output layers to handle unseen vocabularies. Even so, it solves tasks with new linear heads. Soft Language Prompts for Language Transfer replaces language adapters with trained prompt vectors and keeps conventional task modules. It also reports that language modules fail below a data floor (Urdu Wikipedia). That finding constrains us: we must measure, not assume, how much unlabeled text an adapter needs. mmT5 builds language modules into pre-training and shows that shared weights drift away from frozen modules unless parts of the model are frozen. This tells us to keep the backbone and prediction head frozen.

The cloze line keeps the head but uses a monolithic model. LM-BFF established the few-shot value of cloze reformulation in English. Enhancing Cross-lingual Natural Language Inference by Soft Prompting with Multilingual Verbalizer, known as SoftMV, maps the blank to label words in many languages at once. It does so on a dense XLM-R with no language-specific parameters, and its augmentation step depends on bilingual dictionaries. Zero-shot Cross-lingual Transfer of Prompt-based Tuning with a Unified Multilingual Prompt, known as UniPrompt, splits XLM-R's lower layers into a template branch and a context branch. It initializes soft label tokens from English blank-position representations and is evaluated only on six high-resource languages. Prototypical Verbalizer for Prompt-based Few-shot Tuning, known as ProtoVerb, replaces label words with learned class vectors, but only in English.

None of these trains a per-language module whose output-side component grounds the cloze head. That component is our key ingredient. The Obscure Limitation of Modular Multilingual Language Models constrains the evaluation: results that assume the language is known in advance overstate real-world performance, so we also test a setting without that assumption.

Research Program

The backbone is XLM-R in base and large sizes, with all pretrained weights frozen, including the masked-prediction head. For each target language we train a MAD-X-style language adapter plus an invertible output adapter on unlabeled target text using masked prediction. Corpus size will be swept deliberately, from a few thousand sentences upward, to locate the data floor. A task adapter stacked on the English (or Indonesian) language adapter is trained on source-language labels with a cloze template. At test time we swap in the target-language adapters and leave everything else unchanged.

Label words may be split into several subwords, which is the vocabulary-starvation problem. We therefore score each label word $w_y$ for class $y$ as follows:

$$s(y \mid x) = \frac{1}{|w_y|} \sum_{j=1}^{|w_y|} \log p_{\ell}\big(w_y^{(j)} \mid \text{template}(x)\big)$$

Here $|w_y|$ is the number of subword pieces in the label word, $w_y^{(j)}$ is its $j$-th piece placed at the $j$-th blank, and $p_\ell$ is the frozen head's distribution when the language-$\ell$ adapters are active. The prediction is the class with the highest $s$.

We compare three label-word choices: English words, target-language words, and the sum of both scores. We also add a few-shot variant in which the task adapter receives $k$ target-language labels, with $k$ from 1 to 32, so the comparison spans the reported 15-sample pivot.

Two further pieces complete the program. The first is a fragmentation analysis: for each language we measure subword pieces per word under the shared vocabulary and correlate it with the cloze-over-linear-head gain. The second is a routing test that drops the known-language assumption. Each adapter is itself a masked-prediction model, so we can choose the adapter with the lowest masked-word perplexity on the input. This replaces an external language identifier with one already built into the modules.

Baselines

The naive reference is dense XLM-R fine-tuned with a linear head. For the cross-lingual NusaX setting, Cross-lingual Few-Shot Learning on Unseen Languages reports an average F1 of 64.92 on unseen languages, which made it the strongest system in that study. In-context learning with larger generative models reached roughly 47–50 F1. As an upper reference, NusaX: Multilingual Parallel Sentiment Dataset for 10 Indonesian Local Languages reports about 80% average macro-F1 for fully supervised XLM-R-Large and IndoBERT-Large. We rerun the dense baseline ourselves with the same seeds, splits, and tuning budget, rather than relying only on the quoted number.

The strongest modular competitor is MAD-X itself: identical language adapters and task adapter, but with a linear classification head. This isolates the head choice while holding capacity fixed. MAD-X reports gains of more than 5 F1 on NER across 16 languages. For NusaX sentiment, we found no reported result on Lacuna for MAD-X.

The strongest cloze competitor is SoftMV on dense XLM-R. On XNLI it reports a 4.4-point average gain over the best baseline at 1-shot and an 8.9-point gain at 8-shot. We run it with its four soft prompt tokens and its multilingual verbalizer. Where no bilingual dictionary exists, we drop the code-switching augmentation and report that change explicitly.

A ProtoVerb-style prototype head on the same adapters tests whether discrete words matter at all. A UniPrompt run on MARC serves as a high-resource sanity check: it reports 46.61% average accuracy at 32 shots, against 42.59% for vanilla fine-tuning.

Two ablations isolate the key ingredient. In the first, the target-language adapters are active in the network body but the English invertible adapter stays at the output. In the second, cloze runs on the dense model with no adapters. All arms share the same unlabeled text, labeled examples, number of trainable parameters per task, and hyperparameter search budget.

The proposal beats the field if the full system exceeds MAD-X with a linear head at $k \le 16$ on unseen NusaX languages by more than seed variance. It must also exceed dense SoftMV-style cloze on those same languages. Finally, the output-adapter ablation must account for a substantial share of the gain.

Why not just…?

Why not just run cloze prompting on dense XLM-R, as SoftMV does? For high-resource languages this may be enough, and we expect only small differences on XNLI's major languages. The evidence that it falls short elsewhere is the Balinese label-translation result: a dense head simply has no usable representation of unseen label words. The dense-cloze arm and the sign-flip test measure this directly. If dense cloze matches our system on unseen languages, the core claim is wrong.

Why not just use MAD-X with its linear head? MAD-X already supplies per-language capacity. The question is whether a classifier learned from scratch on few labels wastes it. LM-BFF's large few-shot gains, and the 15-sample pivot below which fine-tuning loses its edge, suggest it does. The only difference between the MAD-X arm and ours is the head, so this comparison settles the question directly. We expect the gap to shrink as $k$ grows, consistent with the known saturation of prompting at larger data sizes.

Why not just use language-free prototypes (ProtoVerb or nearest-neighbour classification) and skip label words entirely? This is a real threat, because prototypes sidestep vocabulary starvation entirely. Against it, Nearest Neighbour Few-Shot Learning for Cross-lingual Classification needs transductive centering to remove language offsets and gains only +0.6 on XNLI. Prototypes fit in English adapter space also have no guarantee of surviving an adapter swap. We run prototypes on identical adapters. If they match word-based cloze, we will report that the benefit comes from the adapters and the frozen backbone, not from the word-prediction head.

Why not just continue masked-prediction pre-training of the whole model on target text, then prompt? This is the most serious alternative, since it also grounds the head. It costs a full model copy per language, and it risks overwriting English task knowledge, which is the kind of drift mmT5 documents. We run it at matched unlabeled data and report both accuracy and source-language retention. If it wins on a single language, the honest conclusion is that modularity buys efficiency and multi-language serving rather than accuracy.

Why not just translate? For the NusaX local languages, the dataset authors note that reliable translation systems largely do not exist. On MARC, UniPrompt reports translation-based prompting at only 34.83% at 32 shots. We include translate-test only for XNLI languages where translation is available.

Proposed Experiments

The primary benchmark is NusaX three-class sentiment, using its 500/100/400 splits, evaluated on the 8 languages absent from pre-training. We report macro-F1 averaged over 5 seeds at $k \in \{0, 1, 2, 4, 8, 16, 32\}$ target labels. XNLI's low-resource languages (Swahili, Urdu) test whether the effect holds for languages that were seen in pre-training but are underrepresented. MARC serves as the high-resource control, where we predict little advantage.

Three checks are decisive.

The first is the sign flip. Under dense cloze, target-language label words should score below English label words on unseen languages, reproducing the published finding. With target adapters active, they should score at or above English. If no flip appears, the grounding mechanism is not operating.

The second is the output-adapter ablation. Removing only the target invertible adapter at the output should erase most of the gain of cloze over the linear head. If the gain survives, the improvement comes from body-level capacity, not head grounding.

The third is a set of dose-response curves. The cloze-over-linear gain should shrink as $k$ grows past roughly 15. It should rise with each language's subword pieces per word, and it should fall toward zero as the adapter's unlabeled corpus shrinks below the floor reported for soft language prompts.

Finally, perplexity-based adapter routing should come within a small margin of known-language routing. If it fails, that failure shows the method still inherits the real-world limitation of modular models, and we will report it as such.