Spending a Scarce Vocabulary Budget Where the Model Looks

Multilingual language models break words from low-resource languages into many short, meaningless pieces, because their shared vocabulary was built mostly from English and other large languages. A common fix is to add new whole-word tokens for the target language and initialize their embeddings from existing ones. Each added token, though, needs data to learn a good embedding, and low-resource languages have little data. That forces a choice of which words get their own token. Today that choice is made by frequency. We propose making it with gradient-based saliency, the standard way of asking which input words a trained model's prediction actually depends on. The vocabulary budget would go to words that are both important to the task and badly fragmented.

In two sentences: fine-tune a model on the small labeled task set, use gradient attributions to find words that carry the prediction but are shredded into many subword pieces, then give those words dedicated tokens through embedding-initialization methods from tokenizer transfer before fine-tuning again. The claim is that at a fixed, small budget of new tokens, this beats frequency-based selection on named entity recognition (NER), part-of-speech tagging, and sentiment for African languages. The comparison is designed so that the result, whichever way it goes, says whether saliency carries information that frequency does not.

Core Claim

Subword fragmentation is not equally harmful everywhere. It matters most for the few words a task hinges on, such as named entities, sentiment-bearing words, and inflected verbs carrying agreement morphology. These words are often rare, so frequency-based vocabulary construction skips them. Gradient attribution identifies them directly, and it can be computed from a few thousand labeled examples. Our hypothesis is that attribution-selected vocabulary yields larger task gains than an equal number of frequency-selected tokens. We further predict that the gain will sit specifically on test words that are salient and over-fragmented, and will vanish when saliency scores are shuffled among words of equal frequency.

What Prior Work Gives Us and Leaves Open

Two papers motivate using saliency to allocate scarce cross-lingual resources. SoGo uses token-level loss gradients to pick which English words to replace with dictionary translations for zero-shot spoken language understanding. MVMLT does the same for classification and dialogue state tracking. Both find that substituting only the top 1% most salient keywords beats random substitution. This is direct evidence that saliency concentrates useful signal in very few words.

Both methods, however, operate on the source-language side. Both depend on bilingual dictionaries that low-resource and agglutinative languages often lack. Neither touches the target language's tokenization. Our proposal moves saliency to the target side and uses it to decide what the model's input units should be. No dictionary is needed.

The method borrows from two other papers. Efficient Low-Resource Language Models Using Tokenizer Transfer shows that new-token embeddings can be initialized cheaply. Orthogonal Matching Pursuit (OMP) writes each new embedding as a sparse combination of existing ones. Fast Vocabulary Transfer (FVT) averages the embeddings of a word's old pieces. OMP plus brief monolingual fine-tuning reached the best median MultiBlimp accuracy (0.855) and normalized log-perplexity (1.684) among the transfer methods tested. That work transfers whole vocabularies and selects nothing by task relevance. We reuse OMP and FVT as the initialization step and study the selection question it leaves open.

Discretized Integrated Gradients (DIG) is an attribution method that integrates gradients along paths through real vocabulary embeddings rather than a straight line through meaningless regions. It reduces word-approximation error by 30–50% relative to standard Integrated Gradients. This constrains our design. Vanilla gradients, the kind SoGo and MVMLT used, may misjudge saliency when confident models saturate. We therefore treat the choice of attribution method as an ablation rather than a fixed component.

GradSim is the closest use of gradients to fight low-resource data scarcity. It groups languages by the similarity of their classification-layer gradients and reports Swahili sentiment F1 rising from 58.18 to 63.05. It operates at the level of whole languages and the output layer. Our proposal operates at the level of individual words and the input layer, and the two are complementary.

NGLUEni constrains our ambitions. Continued pretraining on 260M Nguni tokens produced large gains, including more than 20% improvement in zero-shot transfer to isiNdebele and Siswati. Our method targets the regime where such corpora do not exist, so we compare against it rather than claim to replace it.

Research Program

The work has three parts. The first is a diagnostic study that must succeed for the rest to matter.

Using AfroXLMR-large and XLM-R-large fine-tuned on each task, we measure how attribution spreads across the pieces of fragmented words. Attributions are computed with vanilla gradients, Integrated Gradients, and DIG. We sum the scores of a word's pieces to get word-level attribution. We then ask whether three things hold: task-salient words are disproportionately over-fragmented relative to their frequency, model errors cluster on such words, and saliency rank is only weakly correlated with frequency rank. If saliency and frequency rank nearly coincide, the proposal reduces to frequency selection, and we would report that as the finding.

The second part is the selection rule. For each word type $w$ in the labeled training set, let $a(w)$ be its word-level attribution magnitude averaged over occurrences, and let $f(w)$ be its count. Let $n(w)$ be the number of subword pieces the original tokenizer produces for it. We score

$$s(w) = a(w)\, f(w)\, \big(n(w) - 1\big),$$

which approximates the total attribution mass that would be carried by a single token instead of $n(w)$ pieces. The factor $n(w)-1$ counts the positions saved by merging the word, and it is ablated. The top $K$ words by $s(w)$ are added as new tokens, initialized with OMP and adapted with brief continued pretraining on a small unlabeled corpus of about 10MB, the Goldfish-scale setting. The model is then fine-tuned again on the same labeled data. Attribution uses only training data, and all choices are tuned on development sets.

The third part is analysis. We split test-set words into four groups: salient and fragmented, salient only, fragmented only, and neither. Gains are then attributed to these groups, separately for words seen and unseen in the training set.

Baselines

The naive reference is the unchanged model, AfroXLMR-large or XLM-R-large, fine-tuned on the task with its original tokenizer. As the published anchor, AfriSenti reports AfroXLMR-large at 71.2 macro-averaged F1 on AfriSenti. For NGLUEni tasks, the file reports only relative gains, so no reported absolute result was found on Lacuna for those configurations. All baselines are rerun under our exact splits, seeds, and tuning budgets.

The strongest competitor on the precise question is task-corpus frequency selection. It adds the top $K$ fragmented words ranked by $f(w)\,(n(w)-1)$, using the same labeled corpus, the same OMP initialization, the same continued-pretraining tokens, and the same fine-tuning budget. It differs from our method only in omitting $a(w)$, so it isolates exactly our ingredient. We count our method as beating it if its mean gain over the naive model is larger across five seeds at $K \in \{500, 2000\}$, with a paired bootstrap confidence interval excluding zero on at least most task–language pairs.

A second strong competitor is full tokenizer transfer in the style of Efficient Low-Resource Language Models Using Tokenizer Transfer. It adds a complete target-language vocabulary learned from the unlabeled corpus, with the same OMP initialization and continued-pretraining budget. This tests whether careful selection beats simply adding everything when data cannot support many new embeddings.

We also include Nguni-XLMR from NGLUEni as a reference with far more data, labeled as such rather than as a fair competitor.

The decisive ablation is frequency-matched shuffling. Saliency scores are permuted among words within the same frequency and fragmentation bins. This preserves every statistic except the saliency-to-word assignment. If shuffled selection matches our method, saliency adds nothing. Further ablations swap DIG for vanilla gradients and for Integrated Gradients, drop the $n(w)-1$ factor, and swap OMP for FVT.

Why not just…?

Why not just pick words by frequency, as every tokenizer does? This is the real threat, because rare words are both under-represented in vocabularies and hard to learn embeddings for. The task-corpus frequency baseline and the shuffled-saliency control settle the question directly. The diagnostic study tells us in advance whether saliency and frequency diverge enough for selection to matter. If they do not diverge, we report that frequency suffices.

Why not just continue pretraining on more target-language text? NGLUEni shows this works when hundreds of millions of tokens exist. Our premise is that for languages like Siswati or Xitsonga, often only megabytes are available. We therefore match continued-pretraining budgets across all arms and report Nguni-XLMR only as a ceiling. If our 10MB variant closes a meaningful fraction of the gap to that ceiling, that is the useful result.

Why not just use a byte-level model and avoid tokenization choices altogether? Byte models such as Nguni-ByT5 avoid fragmentation by design, but they make sequences longer and still needed large Nguni pretraining. We include ByT5-large fine-tuned on the same labeled data as an additional reference. This reference should not be confused with the naive baseline, which is the fine-tuned AfroXLMR-large or XLM-R-large with its original tokenizer.

Why not just use saliency for code-switching augmentation, as SoGo and MVMLT do? That route needs bilingual dictionaries. MVMLT itself found weak gains for distant language pairs. For Nguni and many AfriSenti languages, dictionary coverage of inflected forms is the very bottleneck. Our method needs only the target-language labeled set already used for fine-tuning.

Proposed Experiments

The main grid covers two kinds of benchmark. From NGLUEni, we use NER (MasakhaNER, SADiLaR NER), part-of-speech tagging (MasakhanePOS, NLAPOST), and news classification for isiZulu, isiXhosa, isiNdebele, and Siswati. From AfriSenti, we use the smallest languages, Xitsonga at about 1.2k tweets, alongside the largest, Hausa at over 22k tweets, to test how gains scale with labeled data. Metrics are entity-level F1, tagging accuracy, and macro F1. Token budgets are $K \in \{500, 2000, 8000\}$ with five seeds each. Grammatical sanity is checked with MultiBlimp, to ensure that selected tokens do not damage general language modeling.

Three decisive checks determine whether the proposal is right. First, at small $K$, attribution selection must beat task-frequency selection, and the shuffled-saliency control must fall back to frequency-level performance. Second, the gain must concentrate on salient, over-fragmented test words. Gains spread uniformly would suggest a generic regularization effect rather than the hypothesized mechanism. Third, the advantage should shrink as $K$ grows and the selection problem loosens. It should be largest in the lowest-data languages, Xitsonga and Siswati, where the budget is tightest.

A negative result on the first check is informative in its own right. It would mean that frequency already captures what task saliency sees in low-resource vocabularies, which is worth knowing before anyone builds more elaborate saliency-guided adaptation.