Language models carve text into subword pieces using a vocabulary fixed in advance. In specialized fields, that vocabulary fits badly: a general tokenizer splits "pneumothorax" into four meaningless fragments. The standard fix is to add whole domain terms as new tokens and train their embeddings, the vectors that represent each token. That fix assumes the domain text can be collected in one place. In hospitals it usually cannot. Federated learning is the setting where the model travels to each institution, trains locally, and only parameter updates come back to a server.
Inside federated learning, adding domain tokens collides with two privacy facts. First, the gradient sent back for the embedding table reveals which tokens a client used. Second, a single-token clinical term can then be read off verbatim. We propose to expand the vocabulary differently. Each new term's embedding is computed, not stored as a trainable row. It comes from the model's own frozen internal reconstruction of the word from its subword fragments, passed through one small shared matrix. Clients train only that matrix and a low-rank adapter. They never send a gradient indexed by token identity. The goal is the domain benefit of a new vocabulary at the privacy and noise cost of a fixed-size update.
Core Claim
Two findings motivate this. Privacy Leakage in Federated Learning in Radiology Reports trained a small transformer across six simulated hospital clients and attacked it with a malicious server. The domain tokenizer RadBERT leaked clinical terms at an "18.1% recall rate", against 12.5% for GPT-2 and 9.4% for LLaMA-2. Single-token medical terms are recovered in one step. Recovering Private Text in Federated Learning of Language Models (FILM) showed something about embedding-layer gradients. An honest-but-curious server can extract the exact bag of words in a batch from them and then reorder that bag into sentences. Its defense is to freeze the embedding layer, which forbids vocabulary adaptation outright.
A third paper supplies the method. From Tokens to Words: On the Inner Lexicon of LLMs found that large language models rebuild whole-word representations at the last sub-token of a multi-token word. In LLaMA-2-7B this peaks around layer 13. A linear map fitted with Procrustes alignment, the best rotation between two vector sets, turns those hidden states into usable new token embeddings without fine-tuning. That cut processed tokens by 10.5% to 14.5%.
Our claim is that this reconstruction can serve as the parameterization of new tokens in federated training. A new term $t$ with subword pieces $s_1,\dots,s_k$ gets input and output embeddings
$$e_{\text{in}}(t) = A_{\text{in}}\, h_L(s_1,\dots,s_k), \qquad e_{\text{out}}(t) = A_{\text{out}}\, h_L(s_1,\dots,s_k).$$
Here $h_L$ is the frozen base model's hidden state at layer $L$ on the last piece. $A_{\text{in}}$ and $A_{\text{out}}$ are shared low-rank-plus-identity matrices, initialized by Procrustes on public text. The original embedding table stays frozen, as FILM recommends.
Two consequences follow, and both are testable.
- Fixed-size updates. The trainable parameter count does not grow with the number of added terms. This sidesteps the noise explosion that Training Large-Vocabulary Neural Language Models by Private Federated Learning for Resource-Constrained Devices documents. There, differential-privacy noise sized to a huge embedding table drowns rare words.
- No direct bag of words. No transmitted gradient has a row per token, so FILM's bag-of-words extraction has nothing to read directly.
We do not claim this is leak-free. The gradient of $A_{\text{in}}$ is a sum of outer products with hidden-state vectors $h_L(t)$. The server can precompute those vectors for every public candidate term. An attacker can therefore project an update onto known candidates to test which terms were present. Measuring that residual channel is a central part of the work, not a footnote.
Research Program
The system has three stages.
Candidate selection. We build the candidate set from public biomedical text, namely PubMed abstracts and public glossaries. Hospitals then vote on which candidates matter locally. Each client scores every candidate on its own notes with an AdaptiVocab-style saving score, meaning frequency times tokens saved, following AdaptiVocab. Optionally it also scores by the loss-reduction gradient that Gold Panning in Vocabulary (VEGAD) uses to select terms centrally. Clients clip and noise these score vectors and sum them under secure aggregation. The server keeps the top $K$.
This differs from Private Federated Discovery of Out-of-Vocabulary Words for Gboard in two ways. Gboard discovers strings character by character and recovered only 16.8% of out-of-vocabulary volume at $\epsilon = 0.315$. Our approach scores a public candidate list in one round. That trades open-ended discovery of entirely novel strings for coverage and fewer communication rounds. We quantify that trade on held-out clinical terms absent from the public list.
Embedding initialization. New terms are initialized and parameterized through the inner-lexicon map rather than by sub-token averaging. Gold Panning reports that sub-token averaging does not reliably stabilize learning. FILM's frozen-embedding regime rules out any form that would need a trainable row per term.
Federated adaptation. Federated domain-adaptive pre-training trains $A_{\text{in}}$, $A_{\text{out}}$ and LoRA adapters, with optional differential privacy (DP-SGD) and an exponential moving average on the server. This builds on FDAPT. FDAPT ran federated biomedical pre-training of DistilBERT and came within 1% of centralized training, but it kept DistilBERT's general tokenizer. Our contribution is the vocabulary-changing layer.
The analysis component asks two questions. Is the candidate-projection attack underdetermined when the rank of $A$ is below the number of distinct new terms in a client batch? How much does differential privacy close whatever remains?
Baselines
No vocabulary change. The naive reference is FDAPT itself: the same base model and adapters, trained federatedly with the original tokenizer and frozen embeddings. It shows whether vocabulary change helps clinical modeling at all, beyond shortening sequences.
Federated tokenizer retraining. The strongest existing federated method is "BPE Sample" from Training a Tokenizer for Free with Private Federated Learning. It samples synthetic text from a privately trained model, retrains the tokenizer on that text, remaps embeddings, and continues training. On Reddit it came "within 1% of the 'oracle' tokenizer" in perplexity, whereas a Wikipedia tokenizer caused a "20% perplexity increase". No reported result was found on Lacuna for it on clinical text. We run it on our clinical splits with the same rounds, privacy budget and tuning budget, and with its remapped embedding rows trainable as in the original.
Trainable new rows under differential privacy. The strongest direct competitor for training new rows is Partial Embedding Updates (PEU) from the large-vocabulary private federated learning paper. PEU updates a sampled subset of embedding rows per round. On Stack Overflow it reported perplexity 64.77 with a 16MB payload, against 66.47 and 112MB for full updates. We apply PEU to the same $K$ new clinical terms with matched noise and payload. We also reproduce its Stack Overflow setting to check our implementation.
Centralized upper references. These are VEGAD and AdaptiVocab trained centrally on the pooled clinical data. VEGAD reports BLEU 28.58 on Article QA and keeps 15.20% on GSM8K, where the original model scores 22.10% and indiscriminate expansion falls to "~6-9%".
Ablations. The first ablation replaces inner-lexicon embeddings with sub-token averages under the same shared-matrix training. The second replaces the federated candidate vote with public-only selection.
What counts as winning. Beating the strongest baseline means matching or exceeding PEU and BPE Sample on clinical perplexity and downstream F1 at equal privacy budget. At the same time, clinical-term recall under FILM and the imprint attack must stay at or below the frozen-embedding no-change baseline.
Why not just…?
Why not just use a public biomedical tokenizer with frozen embeddings and skip federation of the vocabulary? This is the most serious threat. PubMed may already contain most clinical terms. We test it directly with the public-only ablation, which uses the same inner-lexicon embeddings and selects terms from public text alone. If it matches the full system on clinical perplexity and token savings, the federated vote is unnecessary. The contribution then shrinks to "inner-lexicon embeddings make frozen-embedding federated training compatible with a domain vocabulary." We would report it that way. A RadBERT-style tokenizer with trainable rows is not the alternative, because that configuration is exactly what produced the 18.1% leakage.
Why not just run BPE Sample? Its own authors note that synthetic text is less coherent than real data. A model that has seen clinical terms only as fragments may not regenerate them reliably. Its remapped embeddings are also ordinary trainable rows, which reopens FILM's bag-of-words channel. Comparing it under FILM on Dischargesum settles both points.
Why not just freeze everything and accept fragmentation, as FILM suggests? FDAPT shows fragmented federated adaptation already works reasonably on biomedical tasks. It may be good enough for accuracy. The case for our method then rests on sequence-length savings, which matter for long discharge summaries, and on terms the inner lexicon decodes poorly. The no-change baseline measures that gap honestly. Tokens2Words reports that about 23% of multi-token words never decode, so we also report results on that subset.
Proposed Experiments
Clinical setting. We split Dischargesum and MIMIC-CXR across six clients, as in the radiology leakage study, and also run a skewed split. The skewed split induces the vocabulary-distribution skew defined by FDAPT, by giving clients different report types. The base models are LLaMA-2-7B, matching the inner-lexicon analysis, and a smaller model for sweeps over privacy budget and rank.
Metrics. We measure:
- held-out clinical perplexity and tokens per document;
- the nine FDAPT biomedical tasks (named-entity recognition, relation extraction and question answering);
- GSM8K retention against VEGAD's 22.10% to 15.20% trajectory;
- clinical-term recall and exact sentence recovery under FILM, the analytic imprint attack, and our candidate-projection attack. The leakage paper reports exact recovery of 42.3%, 42.1% and 39.4% at batch size 64 for RadBERT, GPT-2 and LLaMA-2, and clinical recall of 9.4% for LLaMA-2.
Decisive checks. The first decisive check is the privacy-utility curve. The method must reduce perplexity relative to no-change while keeping clinical-term recall near the 9.4% fragmented-tokenizer level. Trainable-row baselines should climb toward RadBERT's 18.1%. The second is a rank sweep under the candidate-projection attack. If term presence remains recoverable at practical ranks even with secure aggregation and moderate differential privacy, the privacy argument fails and the method reduces to a noise-efficiency trick. Stack Overflow serves as a sanity replication against PEU's reported 64.77 perplexity.
References
- Privacy Leakage in Federated Learning in Radiology Reports: A Comparative Evaluation of Tokenizer-Driven Privacy Risks
- Recovering Private Text in Federated Learning of Language Models
- From Tokens to Words: On the Inner Lexicon of LLMs
- Training Large-Vocabulary Neural Language Models by Private Federated Learning for Resource-Constrained Devices
- AdaptiVocab: Enhancing LLM Efficiency in Focused Domains through Lightweight Vocabulary Adaptation
- Gold Panning in Vocabulary: An Adaptive Method for Vocabulary Expansion of Domain-Specific LLMs
- Private federated discovery of out-of-vocabulary words for Gboard
- FDAPT: Federated Domain-adaptive Pre-training for Language Models
- Training a Tokenizer for Free with Private Federated Learning