When hospitals, universities or law firms want a translation model that knows their terminology, they cannot pool their documents. Federated learning sends the model to each institution, trains it there on local data, and averages the returned updates on a server. The cheap way to do this today is to freeze the big pretrained network and exchange only small adapters, which are low-rank add-on layers inserted between frozen ones. This freezes the one part of the model where domain terminology most directly lives: the vocabulary matrices. These are the input embedding table and the output softmax table, with one row per token. In a multilingual model like mBART-50 they hold roughly a quarter-million rows, close to half the parameters, so exchanging them densely would undo the communication savings. They are also where a client learns that "clutch" means a purse, or how a multi-word course title should be rendered.
Our claim is that the useful in-domain signal in those matrices sits in a small, client-specific and partly overlapping set of rows. Exchanging those rows is cheap, but only if they are aggregated correctly. If a client sends only its domain-salient rows and the server averages each row over the clients that actually sent it, a federated model can pick up domain terminology at a few percent of dense communication. Plain federated averaging fails at this for a specific reason. Clients that never saw a term contribute a zero update for its row, so a term known to one of twelve clients receives one twelfth of its update. We call this dilution. We expect it to explain much of why off-the-shelf aggregators look interchangeable on domain benchmarks, and we propose to measure it directly.
Core Claim and What Is New
The method has three parts. Each client fine-tunes shared adapters together with a sparse set of vocabulary rows. It chooses those rows by a cheap salience score: the log ratio of the token's local frequency to its frequency in a public general corpus. The score is computed from token counts, not from gradient magnitudes, and is capped at a budget of $R$ rows per round. The client uploads the adapter update, the selected row updates, and per-row token counts. The server aggregates each vocabulary row $t$ as
$$ \Delta e_t = \frac{\sum_{i \in S_t} n_{i,t}\, \Delta e_{i,t}}{\sum_{i \in S_t} n_{i,t}}, $$
where $S_t$ is the set of clients that sent row $t$ this round, $n_{i,t}$ is how often client $i$ saw token $t$, and $\Delta e_{i,t}$ is that client's update to the row. Standard federated averaging instead divides by all participating clients' data and implicitly counts non-senders as zero. Adapters are averaged in the usual way. Rows nobody sends stay frozen. We expect this to limit the general-domain forgetting that in-domain tuning normally causes.
The closest prior work makes clear what is and is not new. Fast Parallel Training of Neural Language Models supplies the core trick. In centralized multi-GPU training it transmits softmax gradients only for rows in the minibatch, the top frequent words, and a random sample, and it reports a 2.1x speedup over asynchronous SGD with no perplexity increase. It assumes identically distributed data on each worker, so dilution never arises. Adaptive Sparse Federated Learning in Large Output Spaces via Hashing trains only the output neurons that a hash of the input embedding retrieves, roughly 30% of them, for next-word prediction on Stack Overflow. Its selection is by input similarity rather than domain salience, it does no adapter-based fine-tuning, and it does not measure terminology. Federated Model Decomposition with Private Vocabulary for Text Classification goes the other way. FEDEVOCAB keeps embeddings fully local, which cuts communication up to 83x and cuts token recovery under gradient-inversion attacks by 80.1%. It covers only input embeddings of small classifiers, and it forgoes any sharing of jargon between clients.
On the adapter side, Communication Efficient Federated Learning for Multilingual Neural Machine Translation with Adapter freezes mBART-50 and synchronizes 8M-parameter adapters with language-family clustering. Only Send What You Need meta-learns per-tensor transmission thresholds. Neither touches vocabulary rows, and neither aggregates at the granularity of a token. Our contribution is the combination of domain-salient row selection, per-row count-weighted aggregation, and adapter-based federated translation, evaluated on terminology rather than BLEU alone.
Research Program
The program has four parts. The first is a measurement study, run before building anything. On a federated split we fine-tune adapters plus dense vocabulary rows and record which rows move, how far, and in how many clients. This tests the premise that in-domain gains concentrate in a few hundred to a few thousand rows per client, and that some rows are shared across clients in the same domain. The small-data results support the premise: a few hundred in-domain sentences gave a 25.5% relative BLEU gain in Domain Adaptation of Statistical Machine Translation using Web-Crawled Resources, and The Multilingual Microblog Translation Corpus found about 1,000 examples enough for substantial gains. Both suggest tuning re-weights a narrow slice of existing knowledge. If row movement instead turns out diffuse, the premise fails and we say so.
The second part builds the method above. The third part addresses a privacy side effect of sending sparse rows. The set of row indices a client sends reveals its vocabulary, which is exactly the leak FEDEVOCAB was designed to close. Following the random-sample component of Fast Parallel Training, we pad each upload with decoy rows drawn from a public frequency distribution. We then measure token recovery under the same gradient-inversion attack (Deep Leakage from Gradients) that FEDEVOCAB uses, at several decoy ratios. The fourth part analyzes forgetting. Because general-vocabulary rows are rarely selected, we test whether sparse row updates preserve out-of-domain quality better than dense tuning. Optum's WMT22 biomedical submission gives the reference point: in-domain tuning dropped general-domain BLEU from 41.0 to 34.74.
The testbed is mBART-50 with adapters, so results are directly comparable to the adapter paper. Clients are built from the file's benchmarks. Europarl provides the parliamentary domain in the adapter paper's 12-client configuration. MAGMATic provides Italian–English academic text, which splits naturally into four university clients. These clients share academic jargon only in part, which makes them the cleanest test of overlap. MAGMATic also reports a multi-word term hit rate of only 53.65% for generic systems, against over 76% for single-word terms. WMT22 ClinSpEn provides biomedical English–Spanish. MTNT provides noisy Reddit text. We report SacreBLEU, single- and multi-word term hit rate, general-domain BLEU retention, bytes uplinked per round, wall-clock time under a simulated commodity uplink, and attack token recovery.
Baselines
The naive reference points are untuned mBART-50 and isolated local fine-tuning of adapters plus vocabulary rows at each client with no synchronization. Together they bound how much federation and tuning add at all.
The strongest existing method is adapter-only federated averaging with frozen vocabulary, in the clustered "adapter-families" form of the adapter paper. It reports 25.76 macro-average BLEU on Europarl against 24.49 for full-parameter federated training, with a 98.7% communication reduction. It is the strongest competitor because it already holds the best quality–communication trade-off in this exact setting. We will reproduce it on our splits, with the same clustering, rounds, local steps and learning-rate search budget. Our method's added communication must be charged against it. Beating it means matching or exceeding Europarl BLEU, and gaining clearly on multi-word term hit rate on MAGMATic and ClinSpEn, while uploading no more than roughly twice the adapter-only bytes.
The communication baselines fall into three groups, each run with the same budget sweep, in bytes or in fraction transmitted. The first is adapters plus dense vocabulary rows under standard averaging, which serves as a quality ceiling within federated training. The second sparsifies that dense update with Top-K magnitude selection, or with MetaSend's meta-learned thresholds from Only Send What You Need. MetaSend reports gains of over 3 SacreBLEU over Top-K on MTNT, with up to 62.3% tensor savings. The third uses hash-selected output rows, following Adaptive Sparse Federated Learning in Large Output Spaces via Hashing.
The FEDEVOCAB-style variant keeps all vocabulary rows local and personalized. It is the strongest privacy competitor. Centralized pooling of all client data is run as a non-deployable upper bound.
Three ablations isolate the key ingredients. The first keeps our salient rows but aggregates them with plain averaging, isolating count weighting. The second replaces salient rows with random rows of equal budget, isolating selection. The third adds server momentum, as in FedAvgM, which FlowerTune found to differ only marginally from FedAvg. This checks that our gains do not merely reproduce outer-optimizer effects.
Why not just…?
Why not just keep the vocabulary frozen and let adapters learn the terminology? This is the real threat. If adapters inside the frozen body can remap ambiguous or multi-word terms, our row traffic is wasted. The evidence that they cannot is indirect. Experiments in Domain Adaptation for Statistical Machine Translation found that adapting the word-mapping component gave the larger gain, 2.53 BLEU. MAGMATic shows multi-word terms are where generic systems collapse, and in-domain adaptation there raised the term hit rate by 7.71 points. The comparison with the adapter-families baseline on multi-word term hit rate settles the question directly. If that gap is under about one point, we abandon the row mechanism and report the negative result.
Why not just keep the vocabulary local, as FEDEVOCAB does? Local rows remove both dilution and the leak, but they also remove sharing. A term that two of the four MAGMATic universities both use is then learned twice from half the data. The decisive test is held-out term accuracy on terms seen by only some clients. Fully local rows should win on terms unique to one client and lose on shared terms. If they also win on shared terms, count-weighted sharing is unnecessary. Local rows also force every client to store and tune a quarter-million-row table.
Why not just apply Top-K or MetaSend to a dense vocabulary update? Magnitude-based selection favors frequent general tokens, whose gradients are largest, rather than rare domain terms. The selection also requires sorting, which costs compute. Agarwal et al. show that such compression overhead can erase the network savings, and MetaSend reports 20–28% wall-clock reductions over Top-K. More fundamentally, either selector still feeds into plain averaging, so dilution remains. The plain-averaging ablation separates these two failure modes.
Why not just pool the data centrally? That is ruled out by regulation in the target settings. It is also not obviously better: MERIT's joint training scored 54.3 against 57.0 for its conflict-aware decentralized variant in Decentralized Instruction Tuning. We run pooling only as an upper bound.
Decisive Checks
The central check is a dilution curve. For each held-out term we bin by the number of clients $k$, out of $N$, whose data contain it. Under plain averaging, the term-hit gain should shrink roughly in proportion to $k/N$, since each row's update is scaled by the share of senders. Under count weighting the gain should be roughly flat in $k$. If both curves look the same, the core mechanism is wrong.
The second check is forgetting. Sparse salient rows should retain clearly more general-domain BLEU than dense rows, measured on Europarl and WMT general test sets against the 41.0 to 34.74 drop reported in the Optum submission. The third check is privacy and cost. With decoys, token recovery should approach FEDEVOCAB's reduction, while upload stays within a small multiple of adapter-only traffic. The fourth checks that the result is not specific to translation: we rerun the dilution curve on FlowerTune's medical split. There, Gemma2-9B-Instruct's reported best average accuracy of 62.25% is the reference, and we test whether the same row-level effect appears in federated instruction tuning.
References
- Fast Parallel Training of Neural Language Models
- Adaptive Sparse Federated Learning in Large Output Spaces via Hashing
- Federated Model Decomposition with Private Vocabulary for Text Classification
- Communication Efficient Federated Learning for Multilingual Neural Machine Translation with Adapter
- Only Send What You Need: Learning to Communicate Efficiently in Federated Multilingual Machine Translation
- Domain Adaptation of Statistical Machine Translation using Web-Crawled Resources: A Case Study
- The Multilingual Microblog Translation Corpus: Improving and Evaluating Translation of User-Generated Text
- Optum's Submission to WMT22 Biomedical Translation Tasks
- FlowerTune: A Cross-Domain Benchmark for Federated Fine-Tuning of Large Language Models
- Experiments in Domain Adaptation for Statistical Machine Translation
- Decentralized Instruction Tuning: Conflict-Aware Splitting and Weight Merging