Progressive token pruning speeds up transformers by dropping the tokens judged least important as text moves deeper through the network. Every published schedule for when and how much to prune has been tuned and checked on English. Languages whose words get chopped into many subword pieces already pay a "token tax": they need more tokens to say the same thing, so they cost more compute and get lower accuracy. A pruning schedule tuned on English could make this worse, for example by discarding fragments of a word's root that only mean something together. For most of the world's languages, nobody could tell, because no labeled test set exists to reveal the damage.
This proposal measures whether progressive pruning imposes a second, hidden tax on over-segmented and poorly understood languages. It then builds a check that predicts per-language pruning damage from raw unlabeled text. We validate that check on the roughly 80 language-task pairs where labels exist, then use it to set per-language pruning budgets for the thousands of languages where labels do not. If the check works, efficient inference can be shipped to low-resource languages with some evidence that it is not quietly breaking them.
Core Claim
We make two claims. The first is that pruning damage varies sharply by language at a fixed compute saving, and that the variation tracks two things, subword fragmentation and the model's basic understanding of the language, rather than resource level alone. The second is that this damage can be estimated without labels by combining two unlabeled-text signals. One is how much the pruned model's outputs diverge from the unpruned model's. The other is how much pruning erodes the model's sensitivity to character order. Neither signal is enough alone.
The reason for needing both is specific. In a language the model does not understand, the full and pruned models both produce near-noise, so they agree trivially and output divergence looks reassuring. The character-order probe catches exactly this failure. It measures whether scrambling adjacent characters shifts hidden states, and it separates intelligible from unintelligible languages (Detecting Languages Unintelligible to Multilingual Models through Local Structure Probes, r = 0.99 with XNLI and Tatoeba accuracy across 44 languages).
Position Relative to Prior Work
Three papers supply the method.
PoWER-BERT scores each token by the total attention it receives and learns how many tokens to keep at each layer. It reports 2.0 to 4.5 times speedups on GLUE with under 1% accuracy loss. It also reports that learned schedules fail to transfer across tasks, which hints that they may fail across languages too.
Adaptive Layer Selection (ASL) waits to prune until the ranking of tokens by attention stops changing across layers. On InfiniteBench with Qwen2.5-7B it reaches 41.8% average accuracy, against 36.2% for FastKV, which prunes at a fixed layer 15. Its evaluation is English-only.
Which Tokens to Use? shows that plain top-K selection by attention matches learned selectors. That justifies auditing simple pruners first.
Three papers motivate the question.
The Token Tax finds that subword fragmentation explains up to 50% of the accuracy variance on AfriMMLU, with up to 18 points lost per extra token per word. It never asks what happens when tokens are removed again inside the network.
XTREME-R documents persistent accuracy deficits for non-Latin scripts.
Multi-task zero-shot performance prediction predicts accuracy from typology, pretraining size and tokenizer fertility. It cuts mean absolute error by 10 to 20%. It predicts accuracy rather than the damage caused by an intervention, and we use it as a competing predictor.
The closest connecting work prunes weights rather than tokens. Kim et al. prune 30% of weights using FLORES-200 parallel pairs and raise XGLM-2.9B's average XNLI accuracy from 42.5% to 44.5%. Ahia et al. find that magnitude weight pruning disproportionately hurts rare words in low-resource translation. Neither paper shortens the sequence. Their findings pull in opposite directions: Kim et al. suggest pruning can steer a model toward cross-lingual pathways, while Ahia et al. suggest it strips the long tail. That conflict is a reason to expect token pruning to have uneven effects across languages. Settling which effect dominates for token pruning is part of what we measure.
Research Program
Stage one: the audit. We take multilingual models fine-tuned on English and apply three pruners at matched savings of 30%, 50% and 70% in attention compute:
- top-K by attention at a fixed layer, as in FastKV;
- PoWER-BERT schedules learned on English;
- ASL's stability-triggered pruning.
The encoders are XLM-R and mBERT, run on XNLI, TyDiQA-GoldP and XQuAD from XTREME-R, which have the long contexts where pruning pays off. The decoders are Qwen2.5-7B and Llama-3.1-8B-Instruct, run on AfriMMLU with long few-shot prefixes.
For each language $l$ we define the pruning tax as $\Delta_l = A_{\text{full}}(l) - A_{\text{pruned}}(l)$, where $A$ is task accuracy. We regress $\Delta_l$ on three predictors: tokenizer fertility, the language's local-sensitivity score, and the layer at which ASL's attention ranks stabilize. That last predictor tests a specific hypothesis: in languages the model barely understands, the ranking never stabilizes, and ASL either never fires or fires on noise.
Stage two: the label-free estimator. On unlabeled text in language $l$ (FLORES-200 monolingual sides, MADLAD-400 samples) we compute two quantities:
- $D_l$, the mean divergence between pruned and full output distributions;
- $S_l$, the drop in local-sensitivity score when the probe is run through the pruned rather than the full model.
We fit a small predictor $\hat\Delta_l = f(D_l, S_l, \text{fertility}_l)$ on part of the labeled languages and evaluate it on held-out languages and held-out language families.
Stage three: per-language budgets. We use $\hat\Delta_l$ to pick, for each language, the most aggressive pruning ratio whose predicted tax stays under a tolerance, such as 1 point. We compare this against a uniform English-tuned ratio and against fertility-normalized budgets.
Baselines
Unpruned model. This is the do-nothing reference and defines $\Delta_l = 0$. For context, XTREME-R reports a 65.3 average for unpruned XLM-R Large.
Strongest existing pruner. On the efficiency side this is ASL, run with its published threshold and with the threshold retuned on English development data. Both pruned systems get the same English tuning budget and the same evaluation sets. Its best reported number is 41.8% average accuracy on InfiniteBench with Qwen2.5-7B, against 36.2% for FastKV. We found no reported result on Lacuna for any token pruner on XNLI, AfriMMLU or XTREME-R, so our audit supplies the first such numbers.
What counts as success for budgets. Our per-language budgets beat a uniform English-tuned ASL or PoWER-BERT schedule if two things hold at equal average compute saving. First, the worst-language tax must be lower. Second, the share of languages exceeding a 1-point tax must be smaller.
Strongest existing predictor. On the estimation side this is a typology-and-fertility regressor in the style of Ahuja et al.. It is trained on the same language split to predict $\Delta_l$ from URIEL distances, pretraining size and fertility.
Ablations. Two ablations isolate the key ingredient: $D_l$ alone, and $S_l$ alone. The estimator succeeds if its rank correlation with true $\Delta_l$ on held-out families clearly exceeds both ablations and the typology regressor. The gain must be largest on languages below the 0.99 intelligibility threshold, which is where we predict output divergence alone fails.
Why not just…?
Why not just translate everything into English first? Translating into English before running the model sidesteps tokenizer inefficiency, but it pays an extra full translation pass, which costs far more than pruning saves. XNLI reports a 3 to 6 point gap even with good translation. For the languages flagged as unintelligible by Clouâtre et al., usable translation systems often do not exist. We report translate-then-run accuracy and cost as a reference on XNLI, but it does not answer the deployment question we pose.
Why not just set the budget per word instead of per token? This is the real threat. If the tax is simply fragmentation, then keeping the same number of tokens per word as in English, or merging a word's subwords before pruning, might remove most of it with no estimator at all. We run both variants as full baselines. If they close the tax everywhere, we report that the fix is fertility normalization and drop stage three. Our hypothesis says they will not close it for low-intelligibility languages, where the problem is the model's poor understanding rather than token count. The audit regression tests this directly: it shows whether local-sensitivity scores explain variance in $\Delta_l$ beyond what fertility explains.
Why not just check agreement between the pruned and full model on unlabeled text? That is $D_l$, and it is one of our ablations. It is cheap and likely good for intelligible languages. Its predicted failure is trivial agreement when both models output noise. If $D_l$ alone matches the combined estimator on held-out languages below the 0.99 threshold, the probe component is unnecessary, and we will say so.
Why not just prune weights instead? Weight pruning as in Kim et al. leaves sequence length and the stored attention keys and values untouched, so it does nothing about the quadratic cost that fertility inflates. It also needs parallel calibration pairs. We include Wanda-style 30% weight pruning only as a cross-check on whether the languages it helps and hurts are the same ones token pruning helps and hurts.
Decisive Checks
Three results decide the project.
Is there a hidden tax? The audit must show that, at 50% attention-compute saving, the tax on languages below the intelligibility threshold, or with high fertility, is significantly larger than on English and French. If the tax is flat across languages, the premise fails and we publish that as a reassuring negative result.
Does the estimator work? Leave-one-family-out rank correlation between $\hat\Delta_l$ and the true tax must beat every predictor baseline. This must hold on XNLI's 15 languages, AfriMMLU's 16, and the 50 in XTREME-R, and the advantage must be concentrated on low-intelligibility languages.
Do estimator-chosen budgets help? Per-language budgets chosen by the estimator must reduce worst-language accuracy loss relative to the per-word budget baseline at equal compute. The per-word baseline is the bar the method has to clear. Matching it is not enough.
We also examine MultiBLiMP minimal pairs (MultiBLiMP) to see whether pruning specifically breaks subject-verb agreement across fragmented words, which would be the long-tail failure reported by Ahia et al., now caused by dropping tokens rather than weights.
References
- Detecting Languages Unintelligible to Multilingual Models through Local Structure Probes
- PoWER-BERT: Accelerating BERT Inference via Progressive Word-vector Elimination
- Adaptive Layer Selection for Layer-Wise Token Pruning in LLM Inference
- Which Tokens to Use? Investigating Token Reduction in Vision Transformers
- The Token Tax: Systematic Bias in Multilingual Tokenization
- XTREME-R: Towards More Challenging and Nuanced Multilingual Evaluation
- Multi Task Learning For Zero Shot Performance Prediction of Multilingual Models
- Pruning Multilingual Large Language Models for Multilingual Inference
- The Low-Resource Double Bind: An Empirical Study of Pruning for Low-Resource Machine Translation
- XNLI: Evaluating Cross-lingual Sentence Representations
- MultiBLiMP 1.0: A Massively Multilingual Benchmark of Linguistic Minimal Pairs