Personal Pronunciation Dictionaries for Federated Conversational Speech Recognition

People do not say words the way the dictionary says them. In casual conversation "going to" becomes "gonna" and "probably" becomes something like "pry". Each speaker also has their own habits of shortening and blurring sounds. Speech recognizers trained on phones often cannot collect this speech centrally, because conversational audio is among the most private data there is. Federated learning is the usual answer: each device trains on its own audio and sends only model updates to a server. So far, federated speech recognition has treated speaker differences as a smooth drift in the neural network's weights or input features, to be corrected or held back. We propose moving the speaker's pronunciation habits out of the acoustic network and into a small, readable, per-device pronunciation dictionary. Each device keeps its own probabilities over the ways it says each word. It shares only noisy counts, which the server pools into a population-level dictionary. That shared dictionary in turn serves as the prior for every device's personal one.

The two-sentence version is as follows. Conversational reductions are discrete and structured, and they are largely shared across the population. We therefore learn them as a hierarchical stochastic dictionary: the population layer is estimated from pooled, privacy-protected counts, and a personal layer on each device is pulled toward the population layer. Our bet is that once pronunciation habits live in the dictionary, the shared acoustic model no longer has to drift in order to fit each speaker. The standard federated tools that hold models close to the global consensus then stop working against the speech they are supposed to recognize.

Core Claim

A stochastic dictionary, or lexicon, assigns a probability $P(v \mid w)$ to each phone string $v$ that a word $w$ might be realized as. Pronunciation Learning from Continuous Speech learns such weights with expectation-maximization (EM, an algorithm that alternates between guessing hidden alignments and re-estimating probabilities from them) over candidates proposed by a grapheme-to-phoneme model. It reports a 1.2% absolute word error rate (WER) reduction over expert lexicons on a weather-query task and significant gains on spontaneous lectures. That work supplies our method. The EM statistics are simple expected counts $n_k(v,w)$ of how often client $k$ realized word $w$ as variant $v$, and counts add across clients. A federated version of the population layer is therefore mechanically easy, and we do not claim it as the contribution.

The contribution is the personal layer and what it does to the rest of the system. Each client forms

$$p_k(v \mid w) = \frac{n_k(v,w) + \alpha_w\, p_g(v \mid w)}{n_k(w) + \alpha_w},$$

where $p_g$ is the population dictionary, $n_k(w)$ is the client's total count for word $w$, and $\alpha_w$ controls how strongly the client is pulled toward the population. We set $\alpha_w$ per word by empirical Bayes: we fit the variance across clients of variant proportions from the aggregated counts.

Frequent words then personalize quickly and rare words stay near the population. This is the discrete, word-level analogue of the regularization that Regularized Federated Learning for Privacy-Preserving Dysarthric and Elderly Speech Recognition applies in a continuous latent space. That paper combines FedProx, a penalty on distance from the global weights, with embedding penalties and distillation toward a frozen global model's outputs. Their approach treats all local deviation as drift, and it yields only about 0.5% absolute WER gains on UASpeech and DementiaBank Pitt.

Our testable prediction is an interaction effect. Without a personal dictionary, strong consensus regularization of the acoustic model should hurt recognition of reduced word tokens specifically. With a personal dictionary, the same regularization should be neutral or helpful, because the dictionary absorbs the discrete part of the speaker's variation.

Research Program

The system is a phone-level recognizer decoded through a word graph built from the lexicon and a language model. We need this hybrid structure to have a dictionary at all. The acoustic model is a self-supervised speech encoder fine-tuned with connectionist temporal classification (CTC, a loss that aligns frame outputs to label sequences) over phones. We initialize it from a public read-speech model so that no conversational audio is centralized. Federated Representation Learning for Automatic Speech Recognition shows that such encoders can even be pre-trained federatedly on speaker-siloed audio with no loss relative to centralized pre-training. That result supports the choice, although it was obtained only on read audiobooks.

Training alternates between two federated steps. In the dictionary step, each client aligns its labeled or self-labeled utterances against its current personal dictionary and computes expected variant counts. It then sends those counts with clipping and Gaussian noise for differential privacy. The privacy concern is real: Privacy Attacks for Automatic Speech Recognition Acoustic Models in A Federated Learning Framework shows that raw updates leak information, and Federated Learning with Differential Privacy for End-to-End Speech Recognition studies the cost of adding such noise to full-model updates. Counts are a far smaller and lower-sensitivity release than full model updates.

To avoid revealing arbitrary novel phone strings, clients report counts only over a candidate set that the server publishes in advance. The candidates are generated from grapheme-to-phoneme variants plus general reduction rules: vowel centralization, final-consonant deletion, and cross-word blending of the kind documented by Improving the Arabic Pronunciation Dictionary for Phone and Word Recognition with Linguistically-Based Pronunciation Rules and A comparison of data-derived and knowledge-based modeling of pronunciation variation.

The candidate set also handles confusability, the problem that a variant added for one word may collide with another word that the client never says. That Dutch study found raw phone-recognizer variants raised WER from 10.7% to 10.9%, while smoothing and confusability pruning brought it to 10.0–10.1%. We use this negative result as a constraint on our design. In our setting, the server scores each candidate by its phonetic edit distance to every other word's variants, weighted by the population word frequency, and prunes it before release. This needs no audio. Each client additionally drops personal variants whose inclusion increases lattice density on its own held-out utterances.

In the acoustic step, clients fine-tune the acoustic model with FedAvg or FedProx. They use alignment targets derived from their personal dictionary, so a speaker who says "pry" is trained toward those surface phones rather than toward the canonical form. At test time the personal dictionary is compiled into the decoding graph on the device. This is a single pass, unlike the client transforms in Federated Acoustic Modeling For Automatic Speech Recognition, which require two-pass decoding to estimate.

Baselines

The naive reference is a single canonical dictionary with an acoustic model trained by plain FedAvg on speaker-siloed conversational data. It tells us how much conversational reduction costs a federated recognizer before anything is done about it.

The strongest federated competitor on this question is personalization of an end-to-end model with bottleneck adapters, as in Parameter-Efficient Transfer Learning under Federated Learning for Automatic Speech Recognition. That work updates 4.01–7.71% of a Conformer and reduces Fleurs EN-US WER from 30.58% to 26.79%. It is the strongest competitor because a network can in principle absorb pronunciation habits implicitly, and because it matches our communication budget. We run it on the same encoder, the same client splits and the same number of rounds. We cap the adapter size so that its per-client upload is no smaller than our dictionary counts plus acoustic update, and we tune both methods over the same learning-rate and rounds grid. We count it as beaten if our system achieves lower WER on reduced tokens at equal or lower upload, and if it forgets less on a held-out canonical read-speech set, since Kan et al. report source-domain interference from adapters.

The second strong competitor is regularized FL in the style of the dysarthric and elderly speech paper. We run FedProx plus output distillation with the same tuning budget, reporting the gain over FedAvg to compare against their roughly 0.5% absolute improvement.

The third is feature-space client adaptation, CAFT and its pre-trained variant from Federated Acoustic Modeling For Automatic Speech Recognition. CAFT-PT reached 15.13% average WER against a 15.83% centralized oracle on that paper's internal cross-silo data. That corpus is not available to us, so we reimplement the method on our splits and report latency alongside WER.

As a centralized ceiling for the dictionary itself, we train the population and personal dictionaries on pooled data without noise. For lexical access on Switchboard, the reference is the Siamese/triplet LSTM of Learning Similarity Functions for Pronunciation Variations, which reports 20.8% WER@1 with a binary LSTM against 59.3% for dictionary lookup.

Two ablations isolate the key ingredient. The first uses the population dictionary only, with no personal layer. The second uses a personal layer with no prior, so that $\alpha_w = 0$.

Why not just…?

Why not just let adapters absorb pronunciation implicitly? This is the real threat. A personalized end-to-end model may learn "gonna" without ever representing it. The evidence against it is indirect: adapter tuning caused source-domain forgetting in Kan et al., which is consistent with pronunciation habits and acoustics being entangled in the weights. The adapter baseline above, with matched upload, settles the question. If adapters match us on reduced-token WER and forgetting, our remaining claims shrink to interpretability, a smaller privacy surface and single-pass decoding, and we will report it that way.

Why not just pool counts into one federated stochastic dictionary and stop? That is our first ablation, and it may capture most of the gain because many reductions are shared across speakers. The personal layer is justified only if it helps on frequent words for speakers whose habits deviate from the population. We test this by splitting the gain according to how far each speaker's hand-labeled variant distribution is from the population's.

Why not just warp features per client? Affine transforms on filterbank features can normalize vocal tract and channel differences, but they cannot delete a phone or substitute a discrete one. CAFT also needs two decoding passes. We include it and also stack it with our dictionary. If stacking is additive, the two methods address different variation.

Why not just add every variant each device discovers? The Dutch VIOS result that raw variants worsen WER from 10.7% to 10.9% already answers this. We nonetheless run a no-pruning variant, because the federated setting may make the confusability problem worse.

Proposed Experiments

The primary testbed is the Buckeye corpus: 40 Columbus talkers, 38.1 hours of conversational interviews and 870,224 hand-corrected phones (The buckeye corpus of speech: updates and enhancements). Each talker is one client, with a chronological 70/30 split of their speech into training and test. The hand labels let us tag every test word token as canonical or reduced. We can then score WER separately on reduced tokens, which is the decisive metric. We can also measure whether a client's top personal variant matches its hand-labeled realization. Because transcribers themselves agree on only 77.8% of vowels, we treat that figure as an approximate ceiling on vowel-level variant recall. No reported ASR result on Buckeye was found on Lacuna, so all Buckeye numbers are new. We add speaker-siloed Switchboard for scale, reusing the lexical-access benchmark above so that our federated personal dictionary can be scored directly against the 20.8% WER@1 centralized LSTM.

The decisive check is a two-by-two design: regularization strength (FedAvg versus FedProx with distillation) crossed with dictionary type (canonical versus hierarchical personal). It is scored on reduced-token WER. The core claim predicts a significant interaction. Regularization should hurt reduced tokens under the canonical dictionary and should not under the personal one. We will test significance with speaker-level bootstrap over the 40 clients.

Secondary checks vary three things. The first is the differential-privacy budget on counts, where we expect degradation to be concentrated in rare words, which the prior should cover. The second is the amount of local data per speaker, where we expect the benefit of the personal layer to grow roughly with $n_k(w)/(n_k(w)+\alpha_w)$, following directly from the shrinkage formula. The third is canonical read-speech WER, to measure forgetting.

A negative outcome on the interaction test is informative too. It would mean that federated consensus regularization is not in fact what suppresses conversational variation, contrary to Gap 3's hypothesis.