Row-Aware Averaging for Multilingual Vocabulary Adaptation Across Low-Bandwidth Compute Islands

A language model with an English-centric tokenizer (the component that splits text into vocabulary pieces) chops Telugu, Amharic, or Santali into many tiny fragments. Speakers of those languages pay more per request, wait longer for replies, and lose most of their usable context. The standard fix is vocabulary expansion. Thousands of new language-specific tokens are added to the model, and the model is trained further on text in the target language. This proposal is about what happens when that training is done the cheap way. We mean several groups, each holding data in its own language, each with its own GPUs, connected by a slow network and synchronizing only occasionally. Our claim is that the fix for tokenization inefficiency creates a specific failure in this setting. The new vocabulary rows are touched by only one island. Standard periodic averaging divides their updates by the number of islands. It also mixes them with "push this token down" signals from islands that never see the token.

We propose support-aware outer synchronization, which is a small change to periodic-averaging training. Each vocabulary row, in both the input embedding and the output layer, is averaged only over the islands whose data actually used that row. Only those rows are sent over the network. We will test whether this closes the quality gap between cheap decoupled training and fully synchronized training for expanded-vocabulary adaptation, and whether it also cuts the part of the communication payload that generic gradient compressors handle poorly. If a single global learning-rate rescaling does just as well, we will report that and the idea collapses to a one-line recipe. The experiments are designed to settle that question.

Core Claim

The claim rests on a periodic-averaging method in the style of DiLoCo. In the DiLoCo scaling-law study, independent replicas ("islands") take $H$ local optimizer steps. Each island then sends its parameter change, the outer loop averages these changes as a "pseudo-gradient" and applies Nesterov momentum. That study kept utilization above 80% with 10× to 100× less bandwidth, but only on English C4 and Dolma, where every island sees roughly the same data and touches every vocabulary row.

Partition the data by language and add new tokens, and that symmetry breaks. Let $\Delta_{k,r}$ be island $k$'s change to vocabulary row $r$ after $H$ inner steps, and let $M$ be the number of islands. Standard outer averaging computes $\frac{1}{M}\sum_k \Delta_{k,r}$. Inner optimizers like Adam normalize update size per parameter, so a Korean-only row that one island moved by a normal amount is shrunk by a factor of $M$. That is an effective learning-rate cut applied exactly to the parameters that vocabulary expansion needs to train fastest.

The output layer is worse. Softmax sends every island a small negative gradient on every output row. So the islands that never see Korean steadily lower the scores of the new Korean tokens, and averaging partly cancels what the Korean island learned.

We replace the average with

$$\bar\Delta_r = \frac{\sum_k s_{k,r}\,\Delta_{k,r}}{\sum_k s_{k,r}},$$

where $s_{k,r}\in\{0,1\}$ marks whether island $k$ "supported" row $r$ during the round. On the input side, support means the token appeared in the island's data. On the output side, the supported set borrows the construction of Xiao et al., who trained RNN language models by transmitting and computing gradients only for output rows in the minibatch, frequent rows, and a small random sample. Here support means the token appeared as a prediction target, or belongs to a shared high-frequency set, or is in a random exploration sample. Rows with $s_{k,r}=0$ are neither computed into the average nor transmitted. We also mask outer momentum for rows untouched in a round, so stale momentum does not keep moving them. We will test a count-weighted variant in which $s_{k,r}$ is replaced by occurrence counts, but we do not expect it to win. The reason is that raw counts reward fragmentation: a language shattered into byte-fallback pieces generates many more hits on shared low-level rows for the same content. This is the distortion that the research file identifies in MERIT-style token-weighted merging.

Relation to Prior Work

Four groups of work bear on this proposal, in different roles.

The first group motivates the problem. Zouhar et al. and Ahia document uneven tokenization across languages and the resulting cost gap of up to 15×. Bibi et al. show that effective context shrinks for fragmented languages.

The second group supplies the adaptation recipe we hold fixed. EEVE initializes each new token's embedding as the mean of the embeddings of the old subwords it replaces, and uses staged freezing to protect English. Nag et al. combine targeted vocabulary augmentation with LoRA continued pretraining for Indian languages. Yamaguchi et al. compare initializations such as FOCUS and CLP+ with LoRA adaptation. All three train centrally, on one GPU or one node. We take their recipes as the inner loop and change only how islands synchronize.

The third group supplies communication-efficient training, and none of it treats vocabulary rows specially. DeMo sends a sparse frequency-domain summary of momentum. SEPARATE sends random low-rank projections of gradients. Both treat all tensors as dense matrices and were tested on English. Feng et al. show, for recommendation models, that embedding tables need their own compression because their dimensions carry no spatial correlation. That result constrains us: we should not expect DCT or low-rank compressors to handle embedding rows well. Agarwal et al. constrain us differently. Compression competes with the backward pass for GPU time, so any method must be cheap on the device. A support mask computed from token IDs is essentially free.

The fourth group is the closest bridge work. MERIT trains isolated branches with zero communication and merges them once with token-count weights. It assumes one shared, static vocabulary. We keep periodic synchronization, allow island-specific rows, and weight by support rather than by token counts. LRPO notes a roughly 60% training slowdown from tokenizer disparity, but it proposes no synchronization change.

Research Program

The regime we target is community-scale adaptation. Several sites each hold text in one language or script family, for example Korean, Arabic, Swahili, and a group of Indian languages from IndicCorpV2. Each site has a node of GPUs, and the sites are joined by links of a few Gbit/s. The 3 Gbit/s figure from the DiLoCo study is our reference bandwidth.

The inner-loop recipe is fixed throughout. We use a shared expanded tokenizer, the union of per-language additions chosen by fragmentation, as in Nag et al. New rows are initialized with subword means, as in EEVE. The main configuration trains the transformer body with LoRA and trains the embedding and output layers in full. In this configuration the vocabulary tables dominate the synchronization payload: tens of thousands of new rows at model width, plus the full output layer, far exceed the LoRA adapters. We also run a full-body variant, in which the body is synchronized with DiLoCo, optionally with DeMo compression.

The work has three parts. The first is diagnostic. Under naive DiLoCo with language-partitioned islands, we measure the per-row effective update size relative to fully synchronized training, and the drift of output scores for tokens an island never sees. This tests whether the predicted shrinkage by $M$ and the output-side suppression actually occur. The second part is the intervention: support-aware averaging, momentum masking, and sparse row exchange. The third part is attribution. Ablations separate the input-side fix from the output-side fix, and support weighting from both a single global rescale and count weighting.

Most models are Qwen2.5-1.5B, which LRPO also uses, so we can sweep $M \in \{2,4,8\}$ and $H \in \{30, 100, 300\}$. We confirm the main result on Llama-3-8B, the model used by Nag et al. Token budgets are on the order of EEVE's 2B tokens.

Baselines

The naive reference point keeps the original tokenizer and trains with LoRA under ordinary synchronized data parallelism. It measures the tokenizer tax that expansion is meant to remove. We report its tokens-per-character ratio and its downstream scores.

The quality ceiling is fully synchronized data-parallel training with the expanded vocabulary, using the same data, same token budget, and same initialization as Nag et al. and EEVE, run on a fast interconnect. Nag et al. report a 2691% summarization improvement on Santali over base Llama-3 in about 3 hours on one A100. EEVE reports English BoolQ of 0.8492 against base SOLAR's 0.8257. We will reproduce the centralized recipe as our ceiling rather than compare against those absolute numbers across setups.

The strongest existing competitor is standard DiLoCo from Charles et al.. It has the same islands, the same $H$, and the same inner optimizer, and we tune its outer learning rate and momentum on a grid of the same size as ours. Its reported results are above 80% utilization at 10× to 100× lower bandwidth, on English only. No reported result was found on Lacuna for DiLoCo under language-partitioned vocabulary expansion.

We also run DiLoCo with DeMo applied to all tensors, including the embedding tables, as the "generic compression" competitor. And we run a MERIT-style variant: zero communication with one final token-weighted merge.

Beating DiLoCo means one of two things at matched communicated bytes and matched wall-clock under throttled bandwidth. Either we recover at least half of the gap between naive DiLoCo and the synchronized ceiling, measured in per-language bits per character, with no English regression beyond the ceiling's. Or we match naive DiLoCo's quality with meaningfully less embedding-table traffic.

Three ablations isolate the key ingredient. One multiplies the outer learning rate on all vocabulary rows by $M$. One applies support weighting only to input embeddings. One replaces support weighting with count weighting.

Why not just…?

Why not just shuffle data so every island sees every language? With identically distributed islands the dilution disappears, and this may well match our method on quality. We will run this configuration and report it honestly. Our method is only needed when data cannot leave its site, whether for data residency, community ownership, or bandwidth reasons. If shuffling is possible and matches the ceiling, the contribution narrows to the partitioned case, plus the byte savings from sparse row exchange. Those savings we still measure, because even with shuffled data many new rows are rare in any single round.

Why not just multiply the embedding outer learning rate by $M$? This is the most serious threat. If languages had fully disjoint vocabularies, a single global rescale would be exactly right for the new rows. It is wrong for shared rows, such as punctuation, digits, and byte-fallback pieces, which would be over-updated by $M$. It also does nothing about output-layer suppression from islands that never see a token. The global-rescale ablation tests this directly. If it matches support weighting, we will recommend it as the simpler recipe.

Why not just train centrally with LoRA on one fast node, like Nag et al.? For one language at 30K sentences, that is the right answer, and our naive and ceiling baselines include it. The case for decoupled training rests on the multi-site, multi-language regime, where DiLoCo's bandwidth argument applies. We will state plainly the scale below which our method brings no benefit.

Why not just compress everything with DeMo or SEPARATE? Those compressors assume dense matrices with concentrated structure. Feng et al. show that embedding dimensions lack the spatial correlation such schemes exploit. More importantly, compression does not fix the averaging rule: a compressed estimate of a diluted update is still diluted. The DeMo-everywhere baseline will show whether its compression error on new rows hurts adaptation more than on body weights.

Why not just train separately per language and merge once, as MERIT does? MERIT's single shared vocabulary and its token-count weighting are exactly what breaks here. Its own analysis, together with WASH (Fournier et al.), indicates that long isolated training drifts out of a shared basin, so a single merge is risky. We include the one-shot merge as a baseline and expect it to fall behind periodic, support-aware synchronization.

Proposed Experiments

The primary metric is held-out bits per character for each language. Unlike per-token perplexity, it is unaffected by vocabulary changes. We also report tokens-per-character reduction, measured bytes sent per outer round, and wall-clock time under links throttled to 3 and 10 Gbit/s.

Downstream evaluation uses IndicGenBench summarization, translation, and question answering for the Indian languages; XNLI for Arabic, Swahili, and German; the Open Ko-LLM Leaderboard for Korean; and the NLI, multiple-choice, summarization, and span tasks from Yamaguchi et al., with their inference speedup measurement. English BoolQ, HellaSwag, PIQA, and ARC-Easy track forgetting.

The decisive checks are three. First, the size of the quality gap between naive DiLoCo and the synchronized ceiling should grow with $M$ for languages that depend most on new rows, and should scale with each row's fraction of supporting islands. If the gap does not track $M$, the mechanism is wrong. Second, support-aware averaging should shrink that gap more than the global-rescale ablation for languages with many shared rows, because that is where the two methods disagree. Third, the output-side mask alone should reduce the measured suppression of the new tokens' output scores on non-owning islands, and that reduction should explain part of the recovered quality.

A secondary analysis compares count weighting with support weighting on the most fragmented languages. The expectation is that count weighting lets those languages dominate the shared byte-fallback rows, which would be a direct, measurable instance of tokenization inefficiency distorting distributed optimization.