A multilingual transformer can carry parts that actively hurt some languages. Shapley Head Pruning showed this for XLM-R Base. It scored each of the model's 144 attention heads with a Shapley value, the game-theory rule for splitting a team's result fairly among its members by averaging each member's added value over all possible sub-teams. Removing heads with negative scores raised XNLI accuracy in all 15 languages and lifted Chinese part-of-speech tagging by 12.4 points. The difficulty is cost. An exact Shapley value requires scoring all $2^{144}$ subsets of heads, so the paper samples random orderings and stops early with statistical bounds. It does this separately for every target language, so the bill grows linearly with the number of languages. That is the same $O(L)$ wall that Lv et al. identify as the main obstacle to scaling multilingual adaptation.
This note proposes that the per-language attribution problems are not independent and should not be solved independently. The 15 or 100 "games" (one per language, each asking how much every head helps that language) share most of their structure. Shapley values are additive, so a language's head scores can be written exactly as the scores of a cheap, precisely estimated shared game plus the scores of a small residual game specific to that language. If the residual is small, as it should be for languages that share vocabulary, then each additional language costs a fraction of the original sampling budget. In two sentences: estimate the Shapley values of a few shared "basis" games once, carefully, by evaluating the same head subsets on parallel sentences in every language. Each language then only needs enough samples to fit its own mixing weights and a low-variance residual, which turns per-language interference maps from a linear-in-$L$ expense into a mostly fixed one.
Core Claim
Let $v_l(S)$ be the utility for language $l$ when only heads in subset $S$ are active. We use the mean log-probability of the gold label on aligned development sentences, a smooth stand-in for accuracy. Let $\phi(v)$ be the vector of 144 head Shapley values of a game $v$. Additivity gives, for any basis games $g_1,\dots,g_r$ and any weights $a_{l1},\dots,a_{lr}$,
$$\phi(v_l) \;=\; \sum_{k=1}^{r} a_{lk}\,\phi(g_k) \;+\; \phi\Big(v_l - \sum_{k=1}^{r} a_{lk}\, g_k\Big).$$
This identity is exact. The useful part is how much it cuts sampling. Under permutation sampling the residual term is estimated from paired marginal contributions on the same orderings, and its variance is reduced by roughly $1-\rho_l^2$, where $\rho_l$ is the correlation between language $l$'s marginal contributions and those of the fitted basis combination. This is the standard control-variate result that ControlSHAP exploits with a Taylor proxy.
Our proxy is not a Taylor expansion of the model. It is the other languages' games, evaluated on shared coalitions. That matters because ControlSHAP's gains fall from 80–95% on smooth models to 15–50% on non-smooth ones, and head masking is a discrete, non-smooth intervention. Cross-language correlation in that setting is an empirical question, not a given.
The claim is therefore falsifiable. If $\rho_l$ is high for most languages, attribution cost per added language drops several-fold at equal accuracy of the head ranking. If $\rho_l$ is low, interference is genuinely idiosyncratic per language. That is a substantive finding about the "curse of multilinguality," and it would justify the per-language cost that Shapley Head Pruning pays.
Research Program
The first component is shared-coalition sampling. Both XNLI and the aligned UDPOS subsets used by Shapley Head Pruning consist of semantically aligned sentences across languages. Each sampled permutation of heads is therefore applied to one mixed batch containing the aligned sentences in all languages, which yields $L$ utilities per prefix. This does not reduce token compute per language by itself. What it buys is pairing: the multi-game control variate is valid only when every language is scored on the same coalitions.
The second component is the basis. We test three options in increasing order of ambition. The first uses a single pivot game, English. The second uses the mean game across languages. The third uses $r$ basis games obtained by a low-rank factorization of the matrix of per-language marginal contributions, collected during a warm-up phase that samples every language. Basis Shapley values are estimated once with a large budget using an efficient single-game estimator. We use Adalina, which comes with an $O(\frac{n}{\epsilon^2}\log\frac{1}{\delta})$ query bound in linear memory.
The third component handles each language. Language $l$ receives a small sampling budget, used to fit $a_l$ by regression on paired marginal contributions and to estimate the residual Shapley vector. An early-stopping rule in the style of the bandit bounds in Shapley Head Pruning stops sampling once each head's sign is settled. Because most residuals should be near zero, most heads should settle quickly.
The fourth component is the new-language regime, which is where the scaling claim actually lives. For a language held out of basis construction, we ask how many coalition evaluations are needed to reach the interference map that full per-language sampling would produce. We also ask whether the learned weights $a_l$ track subword-vocabulary overlap with the basis languages. Lv et al. found such overlap to be the strongest predictor of cross-lingual transfer, ahead of syntactic or geographic distance. If the weights do track overlap, the weights for an unseen language could be predicted before sampling at all.
The pruning output is a per-language binary head mask, applied at inference given the input language. Storage is 144 bits per language, so the deployment cost of being language-specific is negligible. This contrasts with adapter-based fixes that add parameters per language.
Baselines
The naive reference point is the unpruned XLM-R Base fine-tuned on English and evaluated zero-shot. Random pruning of the same number of heads per language is reported alongside it, so that gains cannot be explained by pruning alone.
The strongest existing method is the truncated permutation sampling with multi-armed-bandit stopping from Shapley Head Pruning, run independently per language. It is the only published method that targets exactly this question. It reports an average XNLI gain of +1.59 accuracy points over unpruned XLM-R Base across 15 languages, an average UDPOS gain of 3.4 points across 13 languages, and +12.4 points (24.7% relative) on Chinese UDPOS from pruning 18 heads. We found no reported result on Lacuna for its forward-pass count or GPU time, so we re-run it ourselves and log sentence-level forward passes. The comparison is fair if both methods use the same fine-tuned checkpoint, the same 512 aligned development sentences, the same utility, and the same pruning-selection rule. We compare at two budgets: the budget at which our reimplementation reproduces those gains, and a sweep of matched total budgets. Our method beats it if, at equal total forward passes over all languages, it matches or exceeds per-language pruning accuracy. It also beats it if it matches that accuracy with substantially fewer passes, and we predict at least a two- to three-fold reduction once more than about ten languages are involved.
The strongest off-the-shelf estimator baselines are ControlSHAP, using a first-order Taylor proxy in head-mask gates, and Adalina, each run independently per language at matched budget. These test whether a better single-game estimator already captures the savings without sharing across languages.
The cheapest serious competitor is signed first-order head importance: the gradient of the loss with respect to a continuous gate on each head, computed per language in one backward pass. Shapley Head Pruning reports that Shapley-based pruning beat gradient-based importance pruning. We re-run the signed version specifically, because a signed gradient can in principle flag harm.
Three ablations isolate the key ingredient. The first uses shared coalitions without the control variate, which tests whether pairing alone helps. The second uses a pivot-only basis versus a learned rank-$r$ basis. The third fixes $a_{lk}=1$ for English and zero otherwise, which reduces to "English map plus correction."
Why not just…?
Why not just compute one mask in English and apply it everywhere? Shapley Head Pruning reports that heads pruned for English also help other languages, so this is a real threat. If an English-only mask recovers most of the per-language gains, the per-language problem has little value and neither does our method. The 18-head, 12.4-point Chinese UDPOS result suggests strongly language-specific interference, but suggestion is not evidence. We report English-mask transfer accuracy per language against per-language masks. If the gap is under about half a point on average, we would reframe the contribution as an analysis of which languages deviate from the English game and stop claiming a pruning improvement.
Why not just use the signed gradient with respect to head gates? It costs one backward pass per language and could make all sampling unnecessary. Head removal, however, is a large discrete change, and a first-order estimate at the full model can miss heads whose harm only shows up when combined with others. ControlSHAP's degradation on non-smooth functions points the same way. Our experiments compare the sign agreement and the pruning accuracy of the gradient baseline against high-budget reference Shapley values. If the gradient baseline matches, we would say so and treat the shared basis as unnecessary.
Why not just plug a better single-game estimator, such as ControlSHAP, Adalina or TN-SHAP, into each language separately? These estimators reduce variance within one game, but their cost still grows linearly in $L$. TN-SHAP would also need its surrogate refit per language, and it has only been validated on low-dimensional regression data. Our method composes with them rather than replacing them, since Adalina estimates the basis games. The per-language-estimator baselines at matched budget measure directly how much of the savings comes from sharing across languages.
Why not just add language adapters and skip attribution? Adapters address capacity, but they add parameters for every language and do not reveal which existing parts cause harm. Shapley Head Pruning's gains came with no added parameters. We do not run adapters as a primary baseline, because the question here is cheap diagnosis rather than the best possible multilingual accuracy.
Proposed Experiments
All experiments use XLM-R Base fine-tuned on English, and the XNLI (15 languages, XNLI) and UDPOS (13 languages, Universal Dependencies) setups of Shapley Head Pruning, with 512 aligned development sentences per language for attribution and the standard test splits for evaluation.
Since exact values are impossible for 144 heads, we build reference Shapley vectors by very high-budget permutation sampling, checking stability across five seeds. Estimators are scored on rank correlation with the reference, agreement on head signs, and test accuracy after pruning, all reported as functions of total sentence-forward passes. Every result uses at least three seeds with paired significance tests, which addresses the missing variance reporting that the file notes for this literature.
The first decisive check measures $\rho_l$ for every language against the pivot and the learned bases. This tells us before any pruning whether the savings mechanism is available, and whether $\rho_l$ tracks the subword-overlap measure of Lv et al..
The second is leave-one-language-out. We build the basis from 14 XNLI languages, then measure how many evaluations the held-out language needs to reach 95% of its full-budget pruning gain. We expect far fewer for languages with high overlap, such as Spanish relative to French, than for distant, high-fertility ones such as Thai or Swahili.
The third is a budget sweep comparing accumulated cost across languages. It tests whether total cost grows sublinearly in $L$ while accuracy matches Shapley Head Pruning's reported XNLI and UDPOS gains.
The proposal fails cleanly if $\rho_l$ is low across the board, or if English-mask transfer matches per-language masks. Either outcome would still settle an open question about how language-specific cross-lingual interference really is.
References
- Shapley Head Pruning: Identifying and Removing Interference in Multilingual Transformers
- Efficiently Maintaining the Multilingual Capacity of MCLIP in Downstream Cross-Modal Retrieval Tasks
- Stabilizing Estimates of Shapley Values with Control Variates
- Adalina: Adaptive Linear Approximation for the Shapley Value and Beyond
- Tractable Shapley Values and Interactions via Tensor Networks
- XNLI: Evaluating Cross-lingual Sentence Representations
- Universal Dependencies