Undoing Aligner Confusion in Learned Pronunciation Probabilities

Speech recognizers that use a pronunciation dictionary must know how often each word is said each way. "Butter" may be said with a full /t/ or with a flap, and "probably" may be said in full or reduced to "prob'ly". The standard recipe estimates these frequencies by force-aligning training audio to its transcript, letting the recognizer pick which listed pronunciation best fits each spoken word, and counting the picks. The flaw in this recipe is that the recognizer doing the picking makes mistakes. It systematically mistakes one pronunciation for an acoustically similar neighbor, so the counts measure what the aligner believes, not what speakers said. Reduced and accented forms, the cases pronunciation modeling exists to handle, are exactly the ones most often mislabeled.

We propose to treat this as a communications problem. The true distribution over pronunciations is a transmitted signal. The aligner is a noisy channel with a measurable confusion matrix. The observed counts are the received signal. Recovering the true distribution is then an equalization problem: one inverts a regularized confusion matrix, the same computation a multi-antenna radio performs to separate users. That literature also supplies the practical tools. Truncated series expansions and banded "tridiagonal" approximations replace exact matrix inverses with cheap, stable iterations, and the conditions under which they work are well understood. The payoff is twofold. Debiased probabilities should be closer to human phonetic transcriptions and should give lower word error. The cheap solves also make it feasible to fit a separately regularized lexicon for every speaker or accent group.

Core Claim

Our first claim is that count-based pronunciation probabilities are biased estimators, with the bias determined by the aligner's confusion between pronunciation variants. Our second claim is that a regularized linear inversion of that confusion removes most of the bias. Both claims are testable against manually transcribed phonetic data.

Concretely, let $p_w$ be the vector of true variant probabilities for word $w$. Let $C_w$ be the confusion matrix whose entry $(j,i)$ is the probability that the aligner picks variant $j$ when a human transcriber heard variant $i$. Let $c_w$ be the normalized alignment counts. Then $c_w \approx C_w p_w$ plus sampling noise. The estimator we propose shrinks toward a prior $p_0$, which can be the pooled or canonical distribution:

$$\hat p_w = (C_w^\top C_w + \lambda I)^{-1}(C_w^\top c_w + \lambda p_0),$$

Here $\lambda$ controls how hard the estimate is pulled toward $p_0$. This is structurally the regularized equalizer $(H^H H + \sigma^2 I)^{-1}H^H y$ used in multi-user detection. The regularization term also addresses the data-sparsity problem that Pronunciation and silence probability modeling for ASR identifies for rare words.

Per word, these systems are tiny and trivially invertible. The matrix becomes large when parameters are shared across the vocabulary, which sparse data requires. Variants are generated by phonological rules applied in context, such as flapping, /t/-deletion, and schwa reduction. We therefore parameterize every word's variant distribution through a common vector $r$ of context-dependent rule probabilities, with a Jacobian $J_w$ mapping rule changes to variant changes. Stacking all words gives a single system $A r = b$, with

$$A = \sum_w J_w^\top C_w^\top C_w J_w + \lambda I.$$

Its dimension equals the number of context-dependent rules, potentially thousands. It has to be re-solved for every speaker or accent group when lexicons are adapted. Off-diagonal entries of $A$ arise only when two rules co-occur in the same words and their outputs are confusable. That makes the structure of $A$ an empirical question with a precise consequence. If $A$ is diagonally dominant, a two- or three-term series $A^{-1} \approx \sum_{k=0}^{K} (-D^{-1}E)^k D^{-1}$ suffices, where $D$ is the diagonal of $A$ and $E = A - D$. If it is not, a banded seed is needed.

Relation to Prior Work

The pronunciation side supplies the setting and the baseline recipe. Pronunciation and silence probability modeling for ASR estimates variant probabilities from forced-alignment counts, rescales them so the most frequent variant costs nothing in decoding, and adds word-dependent silence probabilities. We keep its decoding conventions but replace raw counts with debiased ones. Lexical modeling for Arabic ASR: a systematic approach reweights candidate baseforms by expectation-maximization (EM) against acoustics. That approach is the closest implicit treatment of confusion, and we discuss it below as the main competitor. Semi-Supervised Learning of a Pronunciation Dictionary from Disjoint Phonemic Transcripts and Text learns a Bayesian stochastic dictionary by Gibbs sampling. It targets missing words rather than miscounted variants of known words, and it pays sampling cost (37 versus 1.3 minutes per epoch for Gibbs versus its Viterbi-beam approximation) that a closed-form solve avoids.

Discriminative Pronunciation Modeling Using the MPE Criterion trains pronunciation probabilities to minimize recognition errors on lattices. Its objective differs from ours: it optimizes recognition accuracy and does not try to recover what speakers actually said. The two are complementary, and debiased estimates could initialize it. Revisiting Word Neighborhoods for Speech Recognition shows that articulatory-feature distances predict which words get confused. This motivates our expectation that confusion concentrates between variants differing by one articulatory change, and that ordering rules by articulatory similarity should make $A$ near-banded.

The matrix-inversion side supplies the method and its failure analysis. Large-Scale MIMO Detection for 3GPP LTE: Algorithms and FPGA Implementations shows that a truncated series ($K=3$) stays within 0.25 dB of exact regularized detection when the matrix is diagonally dominant. On the Low-Complexity, Hardware-Friendly Tridiagonal Matrix Inversion for Correlated Massive MIMO Systems shows that correlation destroys that dominance. It repairs the problem by inverting the tridiagonal part exactly in linear time and using it to seed the series, reaching within 0.3 dB of exact. This tension is the central technical question for us. Confusable variants are exactly what make $A$ non-dominant, and they are also exactly the cases worth fixing. The polynomial-expansion estimators in Low-Complexity Channel Estimation in Large-Scale MIMO using Polynomial Expansion offer a third option: series coefficients chosen from the eigenvalue spread. We do not use the hardware results from these papers. Our interest is in their accuracy-versus-structure findings.

Research Program

The first step is to measure the confusion matrix. The Switchboard Transcription Project (STP) portion of Switchboard has manual phonetic transcriptions, as used by Revisiting Word Neighborhoods for Speech Recognition. We force-align that audio with a multi-variant lexicon built in the Kaldi style of Pronunciation and silence probability modeling for ASR. For each word token, we record the variant the aligner picks and the variant closest to the human transcription. Most words occur too rarely in STP for per-word confusion estimates, so we estimate confusion per rule-in-context and compose it into per-word matrices. To prevent leakage, we split STP by conversation into a part used for estimating confusion and a held-out part used only for evaluation.

The second step characterizes the structure of $A$ on full Switchboard. We will measure three things: the spectral radius of $D^{-1}E$, the fraction of off-diagonal mass inside a band of width one or two under an articulatory-similarity ordering of rules, and how both quantities change when a stronger acoustic model is used. Stronger acoustic models should push $C$ toward the identity.

The third step builds the solvers. These are an exact Cholesky solve, a diagonal-seeded series, a tridiagonal-seeded series, and conjugate gradient. Each is applied both to the global system and to per-speaker systems, where the regularizer pulls each speaker toward the global solution.

The fourth step converts the estimates into decoding graphs using the existing rescaling and silence modeling and measures recognition.

Baselines

The naive reference point is a single canonical pronunciation per word, with no variant probabilities. This is the starting point that Pronunciation and silence probability modeling for ASR improves upon.

The strongest existing method on this question is that paper's count-based variant probabilities combined with word-dependent silence modeling. It reports word error rate (WER) improving from 13.7% to 13.1% on Switchboard and from 6.90% to 6.49% on Wall Street Journal dev93 with a TDNN acoustic model. We will re-run that exact recipe ourselves with the same acoustic model, training data, and decoding beam. The only change in our system is the source of the probabilities, so any difference is attributable to debiasing. Beating it means a WER reduction on Switchboard that is statistically significant under a matched-pairs test, together with a lower divergence from held-out STP human variant distributions. For that divergence measure, no reported result was found on Lacuna for any method, so we establish both numbers ourselves.

The second strong competitor is soft-count EM reweighting in the spirit of Lexical modeling for Arabic ASR: a systematic approach. It is iterated to convergence using alignment posteriors rather than hard picks. It gets the same alignment passes and lattices as our method, and its number of iterations is tuned on the STP split used for estimating confusion.

Two ablations isolate the key ingredient. The first applies the shrinkage toward $p_0$ with no confusion correction ($C=I$). This shows whether gains come from regularization alone. The second uses confusion correction with exact inversion versus series inversion. This shows what the cheap solvers cost in accuracy.

Why not just…?

Why not just run soft-count EM, which already accounts for confusability? This is the real threat, and we take it seriously. EM with posterior counts is a maximum-likelihood deconvolution. Its fixed-point update is a multiplicative iteration that, in principle, converges toward the same debiased answer. Two differences remain. First, EM corrects confusion only through the acoustic model's own posteriors, which carry the same bias that creates the confusion. Our confusion matrix is calibrated against human transcriptions. Second, multiplicative deconvolution is known to converge slowly when the operator is ill-conditioned, and that is exactly when confusable variants matter. Our experiment plots divergence from STP against the number of passes over the data for EM and for the closed-form estimator. If EM reaches the same divergence and WER within a few passes, the confusion-correction claim collapses into "calibrated EM". We would then reframe the contribution as using human-calibrated confusion inside EM.

Why not just solve exactly with Cholesky or conjugate gradient? On a server, a system with a few thousand unknowns solves in seconds, so exact solving is the default we report first. The approximate solvers matter only for per-speaker or per-accent adaptation at scale, and only if they lose essentially no accuracy. The structural measurements in step two decide this. If $A$ turns out to be far from dominant and far from banded, we drop the approximate solvers and report that negative structural finding. That would be a useful finding in its own right, because it answers whether the conditions from the radio literature carry over to this setting.

Why not just use a better acoustic model, or end-to-end recognition, so confusion disappears? A stronger acoustic model shrinks the off-diagonal part of $C$, and we measure how much bias remains as the acoustic model improves. If the bias vanishes with a modern acoustic model, the method is unnecessary there. End-to-end systems drop the explicit lexicon. However, pronunciation training applications need explicit variant probabilities: Mispronunciation detection without nonnative training data covers only 49.5% of real learner error patterns with its candidate generation, and correctly counted variant frequencies are what such diagnosis relies on.

Why not just train pronunciation probabilities discriminatively? The approach of Discriminative Pronunciation Modeling Using the MPE Criterion reduced character error rate (CER) on HKUST from 49.7% to 48.2% with acoustic co-training. It optimizes error rate rather than fidelity to speech, and it requires lattice iterations. We include debiased initialization followed by its update as a combination experiment rather than treating it as a rival.

Proposed Experiments

The decisive first check is purely statistical. On held-out STP conversations, does the debiased estimator reduce divergence from the human variant distributions relative to raw counts and to EM? The test is reported separately for highly confusable rules, such as flapping versus /t/ and schwa deletion, where we predict the largest gains.

The second check is recognition WER on Switchboard and Wall Street Journal dev93, using the matched re-implementation of the count-based baseline described above. Librispeech test_other with 100 hours of training data (baseline 22.9% to 22.5% in the Kaldi paper) serves as a lower-confusion control where gains should be small.

The third check concerns structure and solvers. We plot spectral radius and in-band energy for $A$, then plot the accuracy of diagonal-seeded and tridiagonal-seeded series against $K$ relative to the exact solve. The prediction from On the Low-Complexity, Hardware-Friendly Tridiagonal Matrix Inversion for Correlated Massive MIMO Systems is that diagonal seeding fails on confusable rule sets while tridiagonal seeding recovers near-exact accuracy at low $K$.

The fourth check covers per-speaker lexicons on Switchboard speakers. We compare WER with global versus speaker-regularized debiased lexicons, together with the wall-clock time per speaker of each solver.

Interpretation of the outcomes is straightforward. If debiasing helps divergence but not WER, the contribution is a more faithful lexicon for diagnosis rather than for recognition. If EM matches it, the contribution reduces to calibrating EM with human-measured confusion.