Spoken-Word Pronunciations as Discrete Group Templates

A pronunciation dictionary maps each written word to a short string of sound symbols, such as "fox" → /f ɑ k s/. Building one takes trained linguists, and that cost is why most of the world's languages have no speech technology. This proposal learns the dictionary from transcribed speech. We collect every spoken instance of a word and search for one discrete sound string that all of those instances align to equally well. The string is drawn from a small inventory of sound units shared across the whole vocabulary. That shared inventory is what turns thousands of per-word templates into a lexicon, rather than a bag of unrelated word models.

The idea comes from medical image analysis. There, unbiased groupwise registration builds a population "atlas" by warping all scans into a latent average at the same time, instead of warping everything onto one arbitrarily chosen reference scan and inheriting that scan's quirks. We ask for the same property in speech: a word's pronunciation should not be the pronunciation of whichever speaker happened to be picked as a reference. The new difficulty is that the template here is not a continuous image but a discrete string. To optimize it by gradient descent jointly with a speech encoder and a spelling-to-sound model, we pass the string choice through differentiable categorical sampling. These are relaxations such as Gumbel-Softmax, which replace a hard "pick one symbol" step with a smooth approximation, so that gradients can flow into the probabilities of each symbol.

Core Claim

Treat each word's pronunciation as the discrete barycenter of its spoken instances. A barycenter here is the unit string that minimizes the average alignment cost to all instances, with every instance weighted equally. Choose that string from a spelling-conditioned distribution over a shared unit inventory, and train the whole system end to end. The claim is that this yields a lexicon closer in downstream recognition quality to an expert dictionary than three alternatives: sequential Bayesian induction, pairwise acoustic matching, and cluster-then-vote pipelines. The claim is also that its entries are measurably less sensitive to which speakers are included.

The claim has two parts, and they can be tested separately. The groupwise part says that a template which is not any single instance beats a medoid or reference-based template. The differentiable discrete part says that optimizing the string jointly with the shared inventory and the encoder beats computing a continuous average and quantizing it afterward.

Where This Sits

Three lines of work attack the dictionary bottleneck. None of them does what we propose.

Joint Learning of Phonetic Units and Word Pronunciations for ASR induces a phone inventory and word pronunciations jointly from transcribed audio with hierarchical Dirichlet processes and Gibbs sampling. It is the closest in goal and supplies our main benchmark protocol. Its model is built on MFCCs and HMM/GMM states and uses conjugate-prior sampling, so it cannot fine-tune a modern self-supervised encoder, and its per-word pronunciations are inferred token by token rather than as group templates.

Automatic Text Pronunciation Correlation Generation and Application for Contextual Biasing (ATPC) extracts XLSR-53 layer-15 features and runs pairwise Dynamic Time Warping (DTW, a dynamic-programming method that finds the best monotone time alignment between two sequences) between character-level segments. From these alignments it builds a pronunciation similarity matrix. ATPC motivates our work, because it shows self-supervised features carry enough phonetic information to replace a lexicon partially. It is also the design we depart from: its comparisons are pairwise, post hoc, and produce similarities rather than a canonical entry.

Towards End-to-end Unsupervised Speech Recognition (wav2vec-U 2.0) constrains the design, as a negative result. Removing its phonemizer and training on characters raised LJSpeech error from 14.4% to 64.0%. This tells us raw orthography is too loosely tied to acoustics to serve as the discrete target without an intermediate unit layer.

Grapheme modeling is the "skip the lexicon" answer. From Senones to Chenones shows that tied context-dependent graphemes can beat phone systems on 960 hours of LibriSpeech. Towards End-To-End Speech Recognition with Recurrent Neural Networks shows that CTC can map audio to characters. Both depend on abundant paired data. The Effects of Input Type and Pronunciation Dictionary Usage in Transfer Learning for Low-Resource Text-to-Speech reports that dropping the dictionary hurts in low-resource regimes, which is our target.

From groupwise registration we borrow a principle rather than a specific algorithm. Hui Zhang et al. define the optimal template as the one minimizing aggregate deformation to all subjects. SrvfNet learns a population template and per-curve time warps jointly for 1D functions, but it has no discrete layer at all. Group-Wise Point-Set Registration Based on Rényi's Second Order Entropy warns that reference-free objectives can collapse everything to one point. We take that warning seriously, as described below. Transformation synchronisation enforces consistency among pairwise transforms in closed form. However, DTW warps are neither invertible nor rigid, so that route does not transfer directly, which is one reason we optimize a shared template instead of synchronizing pairs.

The method for passing gradients through the discrete choice comes from Learning Differentiable Grammars for Continuous Data. That work samples discrete grammar rules with Gumbel-Softmax over video feature sequences, but it aligns one sequence at a time and never builds a shared template. Categorical Reparameterization with Denoising Diffusion Models (ReDGE) proposes a multi-step, temperature-free relaxation to escape the Gumbel "temperature trap," but it was only tested on static tasks: Sudoku, MaskGIT image guidance, and binarized MNIST.

The Model

The model has three trained parts:

  • A speech encoder. This is XLSR-53, the same features as ATPC, initialized and optionally fine-tuned. It turns each spoken instance $x_i$ of word $w$ into a feature sequence.
  • A unit inventory. This is a codebook of $K$ unit embeddings $E$, plus a special deletable "empty" unit so that strings can vary in length up to a fixed maximum $L$.
  • A spelling-to-units network. It reads the letters of $w$ and outputs categorical logits over units at each of the $L$ positions, defining a distribution $q_\theta(p \mid w)$ over pronunciation strings $p$.

Training minimizes

$$ \mathcal{J} = \sum_{w} \mathbb{E}_{p \sim q_\theta(\cdot \mid w)} \Big[ \frac{1}{N_w} \sum_{i=1}^{N_w} D\big(x_i, E[p]\big) - \lambda \, \frac{1}{|B_w|} \sum_{x' \in B_w} D\big(x', E[p]\big) \Big]. $$

The symbols are defined as follows:

  • $N_w$ is the number of spoken instances of word $w$.
  • $E[p]$ is the sequence of embeddings for string $p$ with empty units removed.
  • $D$ is a smoothed, differentiable version of the DTW alignment cost between a feature sequence and a unit sequence.
  • $B_w$ is a sample of instances of acoustically nearby but different words.
  • $\lambda$ weights the second term.

The first term is the groupwise cost. Every instance counts equally, and the template is never one of the instances. The second term is a contrastive penalty. It is our answer to the collapse failure noted by Sánchez Giraldo et al.: without it, a single unit repeated everywhere would align cheaply to everything.

The expectation over $p$ is estimated with straight-through Gumbel-Softmax, following Piergiovanni et al. We compare this against ReDGE as the sampler. Whether the temperature trap bites in this long, highly combinatorial string space is an open question named in the ReDGE work, and it is one of our experimental questions.

Because the logits come from the spelling, the model also works on unseen words. For a word with no audio, the pronunciation is the most likely string under $q_\theta$. For words with audio, it is a group-refined string. This mirrors how Lee et al. couple letter-to-sound rules with pronunciations, but it is trained by gradient descent.

Research Program

We will build the model above, then apply the induced lexicon in the three settings where published numbers exist: a hybrid recognizer, contextual biasing, and unsupervised recognition. We will also measure three properties at the lexicon level:

  • Reference bias. Remove or add single speakers and track how much each entry changes, measured as edit distance between the resulting unit strings. Separately, inject instances from an atypical accent and measure how far the templates drift.
  • Order invariance. Shuffle the order of instances and minibatches and check that entries do not change.
  • Phonetic validity. Where an expert lexicon exists for scoring, measure the normalized mutual information between learned units and expert phones after alignment.

These measurements are what the groupwise-registration lens adds. They quantify directly the "template bias" that the imaging literature cares about, but which speech work has not measured.

Baselines

Naive reference points. These are graphemes-as-units (the "do nothing" lexicon) and the expert lexicon (the ceiling). On the Weather Query corpus, Lee et al. report 32.7% monophone WER for the grapheme lexicon and 13.8% for the expert lexicon. We rerun both with our acoustic-model recipe so every system shares it.

Strongest existing induction method. This is Lee et al.'s hierarchical Bayesian joint induction, which reports 17.0% monophone WER and 13.4% triphone WER against a 15.7% triphone grapheme baseline. It is the strongest competitor because it solves the same problem, from the same supervision (transcribed audio, no lexicon), and reports results against both a grapheme lexicon and an expert oracle. We will reuse the paper's train/test split: 10,000 training and 3,497 test utterances. If the corpus cannot be obtained, we will run the identical protocol on a 10-hour LibriSpeech subset, scoring against the expert lexicon distributed with LibriSpeech. For that setting we found no reported result on Lacuna for induced lexicons. Beating the Bayesian baseline means monophone WER clearly below 17.0% under the same recognizer, and closing a substantial share of the remaining gap to 13.8%.

Pairwise baseline. This is an ATPC-style system. For each word we take XLSR-53 layer-15 segments, compute all pairwise DTW costs, pick the medoid instance as the reference, and quantize it with k-means into units. This isolates the effect of a reference template versus a group template.

Two ablations of our own method.

  • Continuous template, quantized afterward. We compute a continuous DTW-barycenter average of the instances, then quantize it with k-means. This removes the joint discrete optimization.
  • Hard alternating optimization. We alternate exact k-means codebook updates with dynamic-programming search for each word's best string. This removes differentiable sampling but keeps the groupwise discrete objective.

Fairness conditions. All systems use the same encoder features, the same sweep over inventory size $K$, the same hybrid recognizer recipe, and matched hyperparameter-search budgets measured in GPU hours.

Why not just…?

Why not just use pairwise DTW, as ATPC does? ATPC shows how far pairwise matching goes. With its context graph on Aishell it reaches 10.3% CER versus 8.9% for a curated lexicon, starting from a 13.8% baseline. But its output is a similarity matrix, not a canonical string, and any medoid choice inherits one speaker's realization. The pairwise-medoid baseline and the reference-bias test settle this directly. If medoid templates prove as stable and as accurate as group templates, our groupwise claim fails.

Why not just cluster each instance into units and take a majority vote? This is the most serious threat, because it is cheap and also symmetric across instances. Voting operates on already-quantized strings, so it cannot move unit boundaries, and it cannot share evidence across words through a learned inventory. With few instances per word, which is exactly the low-resource case, votes are also noisy. We include it as a baseline and vary the number of instances per word from 3 to 50. If voting matches our method at every instance count, we will conclude that the differentiable machinery is unnecessary and report that.

Why not just optimize the strings exactly with hard alternating updates, with no Gumbel or ReDGE? Pronunciation strings are short, so exact search per word is feasible, and a reviewer may reasonably expect hard alternation to do as well. The case for relaxation is narrower than "it's differentiable." Relaxation lets the encoder and the spelling-to-units network be trained through the lexicon choice. We test this with the hard-alternation ablation, both with a frozen encoder and with a fine-tuned one. If the fine-tuned relaxed model does not beat hard alternation, the sampling component should be dropped.

Why not just skip the lexicon and use graphemes? Chenones show graphemes can win when data is plentiful: 3.2% and 7.6% WER on LibriSpeech test-clean and test-other with 960 hours. In low-resource settings, however, the reported evidence goes the other way: Lee et al.'s grapheme lexicon is at 32.7% monophone WER, and wav2vec-U 2.0 collapses when trained on characters. We report grapheme lexicons at every data size in our study, so the crossover point is visible.

Proposed Experiments

Main experiment. Hybrid recognition on the Weather Query protocol (or the LibriSpeech substitute), reported as monophone and triphone WER against all baselines above. For our own method we also report both samplers, Gumbel straight-through and ReDGE with 3 to 9 steps, along with training stability measured as variance across random seeds and sensitivity to temperature.

Transfer to contextual biasing. We substitute our induced similarities for ATPC's on Aishell, keeping its context graph. The target is to improve on its 10.3% overall CER and its hotword F1 of 80.

Low-resource test. On the CommonVoice Tatar and Kyrgyz subsets, where wav2vec-U 2.0 reports phone error rates of 10.1% and 15.2% using an expert phonemizer, we replace the phonemizer with our lexicon and measure how much of that performance survives. The LJSpeech character-collapse setting (14.4% versus 64.0%) serves as a stress test.

Decisive checks. These determine whether the idea stands:

  • The group template beats the medoid on both WER and the leave-one-speaker-out stability measure.
  • The jointly optimized discrete template beats the continuous average quantized afterward.
  • End-to-end fine-tuning through the sampler beats hard alternation.

If the first check fails, the registration framing adds nothing. If the second or third fails, the corresponding component is not earning its cost, and we will say so.