Alignment Shortlists as Shared Communication Masks for Multilingual Translation Training

Multilingual translation models now use very large subword vocabularies, often 128k to 256k tokens, so that low-resource scripts are not shattered into fragments. In data-parallel training, where many GPUs each process part of the batch and then average their gradients every step, the output layer (the matrix that scores every vocabulary entry) and the input embedding are among the largest tensors that must be exchanged. Yet in any single step only a small, predictable slice of that vocabulary matters. The slice is predictable because the target words a sentence can produce are largely determined by its source words. Old statistical machine translation tooling computes exactly this mapping cheaply and offline: word aligners such as FastAlign produce lexical translation tables that list, for each source word, its likely translations.

We propose to use those offline tables to fix, before each step begins, a single set of vocabulary rows that every GPU agrees on. Each GPU then packs only those rows into a dense buffer and averages it with the standard collective operation used by ordinary data parallelism. Because every rank selects the same rows, the buffer stays dense and identically shaped. Because the row set is computed from precomputed statistics on CPU, the GPU spends no time encoding the gradient. Together these two properties target the two documented reasons gradient compression usually fails in practice. The idea in two sentences: statistical alignment tables tell us ahead of time which vocabulary rows a training step will need, so all workers can agree on that row set and run a normal, cheap dense average over just those rows. Combined with batching that keeps each step's target languages narrow, this should cut the dominant share of synchronization traffic in large-vocabulary multilingual models without the encoding cost that sinks most compression schemes.

Core Claim

Three earlier results make this concrete. Vocabulary Selection Strategies for Neural Machine Translation showed that alignment- and phrase-table-derived shortlists of about 600 target words (about 1% of the vocabulary) recover over 99% of the full model's choices and cut single-GPU training time by 25–40%. That paper supplies our selection method, but it never touched multi-GPU communication. It also selected candidates per sentence, which gives each worker a different row set and so rules out dense averaging. Fast Parallel Training of Neural Language Models motivates the communication side. It reported that softmax gradient exchange can take up to 50% of multi-GPU step time, and it sent only rows for batch words, the most frequent words, and a random sample, gaining 2.1x on 4 GPUs. Its row choice used global frequency, not source-conditioned translation statistics, and it ran on recurrent models over asynchronous parameter-server updates rather than synchronous dense averaging. On the Utility of Gradient Compression in Distributed Training Systems constrains the design. Across more than 200 configurations, compression gave end-to-end speedups in only 6. Its explanations were GPU encoding time (about 45 ms for rank-4 PowerSGD on ResNet-50) competing with the backward pass, and sparse payloads forcing a fallback from all-reduce to all-gather.

Our claim is that a step-level, rank-agreed, alignment-derived row set removes both of those failure causes. Paired with target-language-grouped batching, it keeps the communicated fraction of the vocabulary matrices small enough to matter.

Research Program

Before training, we run FastAlign on the training bitext of every language pair. From the output we build a table that maps each source subword to its top-$k$ target subwords. We use the same deterministic data sampler on all ranks, so at step $t$ every rank knows the token IDs of the entire global batch $B_t$, not just its own shard. Each rank computes the same row set on CPU, one step ahead:

$$U_t = \bigcup_{(x,y)\in B_t} \Big( \mathrm{tgt}(y) \cup \bigcup_{s\in x} A_k(s) \Big) \cup F_p,$$

Here $\mathrm{tgt}(y)$ is the set of reference target tokens, $A_k(s)$ is the top-$k$ alignment candidates of source token $s$, and $F_p$ is the $p$ most frequent tokens of the step's target languages. The training loss uses a softmax restricted to $U_t$, in the spirit of the candidate-list training of On Using Very Large Target Vocabulary for Neural Machine Translation. With the loss restricted this way, only the rows in $U_t$ receive gradient. Those rows of the output and embedding matrices are gathered into a contiguous buffer and averaged with ordinary NCCL all-reduce. All other layers synchronize densely as usual.

Rows outside $U_t$ receive no update at that step. Their optimizer statistics are frozen rather than decayed. To prevent slow drift in rarely selected rows, we run a full dense synchronization every $N$ steps.

The quantity that decides whether this pays off is the fraction of the vocabulary matrices that is communicated, $\rho_t = |U_t|/V$, where $V$ is the vocabulary size. With $E$ the share of total parameters in the vocabulary matrices, per-step communication falls to roughly $(1-E) + E\rho_t$ of the dense volume.

Two things threaten this. First, $E$ may be small in a deep Transformer, which is the open question Gap 5 in the research file raises about whether attention and feed-forward layers simply restore the communication bottleneck. Second, $\rho_t$ may approach 1 because a large global batch drawn across 100 languages covers most of a shared vocabulary. The first is measured, not assumed. A 256k-token embedding at width 1024 is about 260M parameters, which is a large share of a sub-billion-parameter model. For the second, our response is a scheduler: each global step draws from a small group of target languages, grouped by script, so the union stays narrow. The research program is therefore three pieces: the shortlist-and-pack synchronization, the language-grouped scheduler, and a measurement study of how $E$ and $\rho_t$ vary with model width, vocabulary size, global batch size, and grouping.

Baselines

The naive reference point is plain synchronous data parallelism with gradient bucketing and communication overlapped with the backward pass, using full softmax. It is the strongest practical competitor in well-connected clusters, as the gradient compression utility study shows. It runs on identical data, model, optimizer, and token budget. Beating it means lower wall-clock time to a fixed validation loss with no worse FLORES-101 spBLEU. No reported result was found on Lacuna for multilingual Transformer translation trained this way on FLORES-101. The file's reference points are the benchmark's own baselines in The FLORES-101 Evaluation Benchmark for Low-Resource and Multilingual Machine Translation: 17.55 spBLEU average for high-resource languages and often below 3 for low-resource ones. We therefore report our own dense baseline as the reference.

The ablation that isolates our key ingredient replaces alignment candidates with the frequency-and-random row choice of the Fast Parallel Training paper. This uses $\mathrm{tgt}(y)$, $F_p$, and a random sample, with the same agreed-row all-reduce and the same row budget $|U_t|$. If alignment candidates do not beat this at equal $\rho_t$, the statistical-translation component adds nothing. A second ablation drops language grouping, to show how much of the gain comes from the scheduler alone. A third uses only the global batch's reference tokens, with no extra candidates of any kind.

The strongest existing compression method is DeMo, from DeMo: Decoupled Momentum Optimization, which transforms momentum with a discrete cosine transform and keeps the top-$k$ components. It reports cutting per-step transfer on OLMo-300M from 637 MB to 7.5 MB (85x) while matching AdamW. We also run SEPARATE from SEPARATE: A Simple Low-rank Projection for Gradient Compression in Modern Large-scale Model Training Process, which uses shared random projections with error feedback and reports halving GPT-2 Medium wall-clock time to target loss. Each gets the same hyperparameter-search budget as our method (for example, DeMo's $k$ and SEPARATE's $\beta$), applied to the vocabulary matrices and to the whole model. Beating them means faster wall-clock time to target loss at equal quality, even where they send fewer bytes, because they pay encoding time and we do not.

For changing how often workers synchronize rather than what they send, the baseline is DiLoCo, from Communication-Efficient Language Model Training Scales Reliably and Robustly: Scaling Laws for DiLoCo. It reports 10–100x less bandwidth with $H=30$ local steps. We also test our method combined with DiLoCo's outer synchronization.

Why not just…?

Why not just rely on overlapped synchronous data parallelism? On fast intra-node links this may well win, and we expect it to: the compression utility study found syncSGD near ideal scaling around 10 Gbps. Our claim is limited to bandwidth-constrained regimes, such as multi-node Ethernet and throttled links, where the file reports communication taking up to 50% (Fast Parallel Training) or up to 92% (SEPARATE) of step time. The bandwidth sweep shows exactly where the crossover lies. If no crossover appears at 10 Gbps or below, the proposal fails.

Why not just use a frequency-based shortlist, which needs no aligner? This is the most serious threat. With subword vocabularies and language grouping, the batch's own reference tokens plus frequent tokens might cover nearly everything the alignments would add. Alignment candidates should matter as hard negatives: plausible wrong translations that the restricted softmax must learn to rank below the correct one. A frequency list misses these for rare words. The equal-budget ablation settles the question. We will report the gap separately on low-resource FLORES-101 directions, where we expect it to be largest. If the gap is null, we will present the contribution as the agreed-row synchronization and the scheduler, not as a hybrid with statistical translation.

Why not just compress gradients with DeMo or PowerSGD? These methods compress further in bytes, but the file documents GPU encoding costs and non-standard collectives as the reasons compression rarely yields end-to-end gains. Our row selection runs on CPU one step ahead and uses dense all-reduce. The comparison of time-to-target-loss across the bandwidth sweep shows which effect dominates. The two approaches can also be stacked, with compression on non-vocabulary layers only.

Why not just use DiLoCo? DiLoCo cuts traffic for every layer, not just the vocabulary matrices. However, it has been validated on i.i.d. pretraining text, and language-skewed workers risk drifting apart before averaging. We treat it as complementary and test the combination. If DiLoCo alone matches the combination at equal bandwidth, the extra value of our scheme is limited to settings where workers must synchronize every step.

Proposed Experiments

The main setting is a many-to-many Transformer translation model with a 256k SentencePiece vocabulary, evaluated on FLORES-101 spBLEU and broken down into high- and low-resource directions. The negative control is a bilingual WMT'14 En-De model with a 32k vocabulary, where $E$ is small and we predict little gain. A WMT'14 En-Fr run links back to the large-vocabulary results of Jean et al. Hardware runs on 8 to 64 GPUs, with inter-node bandwidth throttled to 100, 10, and 1 Gbps. We report bytes per step, $\rho_t$, measured $E$, encoding time, wall-clock time to target validation loss, and spBLEU both at a fixed token budget and at a fixed wall-clock budget.

There are three decisive checks. First, the measured $E\rho_t$ must imply at least a 2x reduction in communicated volume in the multilingual model. If language grouping cannot keep $\rho_t$ low without costing spBLEU, the idea fails. Second, restricted-softmax training must stay within 0.5 spBLEU of dense training. Third, at 10 Gbps or below, our method must beat both dense overlapped data parallelism and DeMo on wall-clock time to target loss. The frequency-shortlist ablation then determines whether the statistical alignment tables earn their place.