Multilingual systems increasingly give each language a small trainable module on top of a shared frozen model. These modules, called adapters, are small bottleneck layers inserted into a pretrained Transformer. When people search for the best shape of such a module (which layers get one, how wide each one is), they usually treat every language as a separate search problem. That makes the cost of the search grow with the number of languages. It also forces the search to rely on whatever little data the target language has, which is exactly where the search signal is weakest. This note proposes to answer, with measurement rather than assumption, whether that per-language search is worth anything. If it is, the note tests whether the answer for a new language can be predicted from a handful of searched "anchor" languages instead of being searched afresh.
The concrete plan is to build a per-language architecture benchmark for language adapters on XLM-R, modeled on how NASBench-style tables made vision search measurable. Over this benchmark we would measure three things. The first is how much accuracy is lost by giving every language one shared, searched architecture. The second is whether cheap scoring rules computed at initialization rank architectures reliably within each language, especially for low-resource and non-Latin-script languages. The third is whether a language's best architecture can be predicted from its typological neighbors. The outcome is useful either way. If the per-language gap is negligible, the field can stop paying for per-language search. If it is real and predictable, search cost drops from one search per language to one search per anchor.
Core Claim
We hypothesize that the best adapter architecture does differ across languages, but along few directions that follow linguistic typology. Under that hypothesis, searching carefully on roughly six typologically spread anchor languages and transferring the result to a new language by typological similarity should recover most of the gain of a full per-language search. It should do so at a small fraction of the cost. We further hypothesize that "zero-cost proxies" (scores computed from a single forward and backward pass at initialization, used to rank architectures without training them) become unreliable precisely in the languages where per-language search is supposed to help most. Both claims can be falsified on the benchmark we build.
What the Literature Gives Us, and What It Leaves Open
Evidence that language-specific structure matters comes from Multilingual Lottery Tickets to Pretrain Language Models. That work scales mBERT to 16 layers and then uses SNIP, a gradient-sensitivity pruning score, to extract a subnetwork per language in minutes. It reports average NER F1 of 84.91 against 84.21 for 12-layer mBERT at slightly lower inference cost (338M versus 341.7M FLOPs). This motivates our question, but it does not answer it. The gain mixes the effect of extra pretraining capacity with the effect of per-language selection. The selection also uses SNIP, which LPZero: Language Model Zero-cost Proxy Search from Zero shows ranks Transformer architectures poorly: Spearman's ρ is 0.173 on FlexiBERT, against 0.748 for LPZero's evolved symbolic proxy. That LPZero result constrains our design. We cannot assume an off-the-shelf proxy is a trustworthy per-language search signal. We have to measure its reliability language by language. EZNAS, which evolves proxy formulas for vision networks, adds a second warning: proxies are noisiest among the top candidates, which is exactly the region a search cares about.
The adapter literature supplies both the search space and the method. MAD-X stacks a language adapter (trained by masked language modeling) under a task adapter (trained on English labels) in a frozen model, swapping language adapters at test time. Its topology is fixed and uniform, at about 8.25M parameters per language. Adaptable Adapters learns which layers receive an adapter using Gumbel-Softmax switches, a differentiable relaxation of on/off choices. It keeps about 13 of 24 layers and raises the GLUE low-data-100 average from 44.87 to 51.28, but it was tested only on English tasks. We reuse its search mechanism as our "expensive but reliable" per-language search. Balancing Fidelity and Plasticity shows, also only in English, that capacity needs vary sharply with layer depth. This suggests a natural space of per-layer allocations, though it says nothing about whether those allocations vary by language.
Two works shape the transfer step. MAD-G generates adapter weights from 289 typological features in URIEL and lang2vec, training 50x faster than per-language MAD-X. It keeps the topology fixed, and it degrades when a whole language family is absent from training. We transfer topology rather than weights, and we treat that family-absence failure as a test condition. Rethinking Performance Estimation in Neural Architecture Search shows that search needs rank preservation, not absolute accuracy, and that the training budget can be cut drastically while preserving rank. We use that idea to make the benchmark affordable.
Heavier machinery bears on the question less directly. AutoMoE searches heterogeneous mixture-of-experts translation models with a trained supernet, but only on bilingual WMT pairs. Memory-efficient NLLB-200 prunes language-specific experts from a 54.5B translation model by routing statistics. Neither asks whether the per-language structure they find is predictable across languages. We stay with encoders and adapters so the ground truth is cheap enough to compute exhaustively.
Research Program
The search space gives each of XLM-R base's 12 layers one of four adapter widths: none, 48, 96, or 384 hidden units. Total parameter count is the deployment constraint, and we sample architectures stratified across parameter budgets so that comparisons are made at matched size.
For each architecture we train one English language adapter and one English task adapter per task. We then train a language adapter of the same architecture for each target language and record zero-shot task accuracy. The architecture is thus shared between source and target within each evaluation, which preserves MAD-X's swapping logic.
We use 24 target languages. They are chosen to span scripts and families, and they include low-resource languages from MasakhaNER and Universal Dependencies 2.7 alongside higher-resource WikiANN languages.
Exhaustive full-budget training is too expensive, so we proceed in two steps. First, following the budgeted-estimation logic above, we find a reduced training schedule whose architecture ranking agrees with full training on a calibration set of 20 architectures in each of 6 languages. We then evaluate about 150 sampled architectures per language under that schedule. Every reported regret is re-checked at full budget for the top candidates.
Three analyses run on this table. The first is per-language regret: the accuracy lost when a language uses the single architecture that is best on average, instead of its own best at the same parameter count. The second is per-language rank reliability: the Kendall τ of SNIP, of a parameter-count baseline, and of an LPZero-style evolved proxy against measured accuracy, broken down by resource level and script. The proxy is evolved only on held-out languages to avoid leakage. The third is predictability: we fit a small model that maps a language's typological vector to a predicted per-layer width profile, trained only on anchor languages. We then test it on held-out languages, including entire held-out families.
Baselines
The naive reference point is standard MAD-X: uniform adapters at every layer with the default bottleneck, plus a parameter-matched uniform variant at each budget in our sweep. The matched variant separates the effect of "searching" from the effect of simply having fewer parameters. Beating it means a positive accuracy difference at equal parameters, averaged over target languages, that survives seeds. We use three seeds, with the threshold set by seed variance. No absolute WikiANN number is given in the file for this configuration. The only reported figure is that MAD-X beats prior baselines by over 5 F1 on average across 16 languages.
The strongest competitor for the efficiency question is one universal searched architecture. This is Adaptable Adapters' Gumbel-Softmax search run once on pooled multilingual masked language modeling data, then given to every language. It is the strongest alternative because it pays the search cost once, exactly as our transfer method aims to, without any per-language claim. It gets the same data, the same total GPU budget as our anchor searches combined, and the same tuning sweep. Our method beats it only if typology-transferred architectures achieve lower mean regret on held-out languages at equal search cost.
The strongest per-language competitor is the lottery-ticket approach: SNIP-guided per-language selection, as in Multilingual Lottery Tickets. The reference number is its NER F1 of 84.91 against 84.21 for mBERT. That result is on a different backbone and search space, so we reimplement SNIP selection within our adapter space rather than compare across papers. We include an evolved Transformer proxy in the LPZero style as the improved version of this baseline. The reported ranking reference for that proxy is Spearman's ρ of 0.748 on FlexiBERT from LPZero. No reported result was found on Lacuna for zero-cost proxy ranking on multilingual adapter architectures.
The expensive upper bound is a full Adaptable Adapters search per target language. Its cost is logged in GPU hours, and its regret defines how much there is to recover.
The ablations isolate the key ingredient, which is typology-guided transfer. One transfers from randomly chosen anchors instead of typologically chosen ones. Another replaces URIEL vectors with representations derived from a few hundred unlabeled target sentences, which tests whether external typology is needed at all given MAD-G's dependence on it. A third uses simple nearest-neighbor copying of the most similar anchor's architecture in place of the learned predictor.
Why not just…?
Why not just search one architecture for all languages and stop? This is the most serious threat. It may well be correct, and the proposal is designed so that it can win. If universal-architecture regret is within seed noise for most languages, we report that per-language architecture search is wasted compute in this regime. That would be a useful negative result, given that per-language selection is currently motivated by gains as small as 0.7 F1 in the lottery-ticket work. The per-language regret analysis settles this directly, and it is reported before any transfer method is evaluated.
Why not just run a zero-cost proxy per language, since it costs minutes? Cost is not the objection; reliability is. LPZero reports SNIP at ρ = 0.173 on Transformer architectures, and EZNAS reports noise among top candidates. Neither measured how reliability changes with a target language's data size or script. Our per-language rank-reliability analysis gives that measurement. If an evolved proxy keeps high τ even on MasakhaNER languages, then per-language proxy search beats amortization on simplicity, and we will say so.
Why not just copy the architecture of the most similar language? That is a reasonable simpler version of our method, so we include it as an ablation rather than argue against it. If nearest-neighbor copying matches the learned predictor, we adopt it as the recommended method. The learned predictor earns its place only by doing better on held-out families, where no close neighbor exists. That is the setting in which MAD-G's typology-conditioned generation is known to fail.
Proposed Experiments and Decisive Checks
The first decisive check is the regret distribution. For each of the 24 languages, we compare the full-budget accuracy of its own best architecture with that of the universal best, at three parameter budgets. We will treat a median regret below seed standard deviation as a stop signal for the transfer half of the project.
The second check is proxy reliability. We report Kendall τ per language for SNIP, parameter count, and the evolved proxy, and regress τ on log pretraining data size and script. A clear drop for low-resource languages would support amortization over per-language proxies.
The third check is held-out transfer. We search on six anchors, predict architectures for 18 held-out languages, and report mean regret alongside total search GPU hours for each method: universal search, per-language proxy, per-language full search, nearest-neighbor, and the learned predictor. We repeat the comparison with entire language families withheld from the anchors.
The result that would confirm the central claim has three parts. Per-language regret is real, meaning above seed noise for a substantial share of languages. Transferred architectures recover most of it. And they do so at roughly anchors-over-languages of the cost of per-language full search, while beating the universal architecture at equal search budget. If instead transfer recovers regret only within families seen among the anchors, we would conclude that typology describes architecture preferences only locally. That would be a narrower but still actionable finding about how to choose anchors.
References
- Multilingual Lottery Tickets to Pretrain Language Models
- LPZero: Language Model Zero-cost Proxy Search from Zero
- EZNAS: Evolving Zero-Cost Proxies For Neural Architecture Scoring
- MAD-X: An Adapter-Based Framework for Multi-Task Cross-Lingual Transfer
- Adaptable Adapters
- Balancing Fidelity and Plasticity: Aligning Mixed-Precision Fine-Tuning with Linguistic Hierarchies
- MAD-G: Multilingual Adapter Generation for Efficient Cross-Lingual Transfer
- URIEL and lang2vec: Representing languages as typological, geographical, and phylogenetic vectors
- Rethinking Performance Estimation in Neural Architecture Search
- AutoMoE: Heterogeneous Mixture-of-Experts with Adaptive Computation for Efficient Neural Machine Translation
- Memory-efficient NLLB-200: Language-specific Expert Pruning of a Massively Multilingual Machine Translation Model