Converting spelling to pronunciation, or a name in one script to another, means learning which chunks of input letters produce which chunks of output symbols. The letters "sh" produce one sound, and "x" produces two. The strongest classical systems for this, grapheme-to-phoneme (G2P) conversion and transliteration, are built as a pipeline. First, an unsupervised aligner (a probabilistic model trained with expectation-maximization, or EM) decides once and for all how each training word splits into letter-chunk/sound-chunk pairs. Then a discriminative decoder (a model trained to prefer correct outputs over wrong ones, using many hand-designed features) learns to produce pronunciations from those fixed splits. This is the same "align, then extract" pipeline that statistical machine translation used for years. In translation it has a well-documented failure: a single wrong word link blocks whole families of valid phrase pairs, and nothing downstream can repair it.
We propose to remove the commitment step. The generative aligner should not hand the decoder one alignment. It should hand over a soft extraction set: every chunk pair it considers plausible, each with a calibrated posterior score. The discriminative learner then chooses, during training, which alignment of each gold pronunciation to learn from. This treats the alignment as a hidden variable steered by the generative posterior. There is a useful formal fact behind the design. For a latent-variable multigram model, the Fisher score, meaning the gradient of the log-likelihood that the Fisher kernel literature turns into features, is essentially the vector of posterior expected chunk-pair counts. So the soft extraction set is the generative model's Fisher information localized to positions in the word. It is not an ad hoc heuristic.
Core Claim
Current discriminative many-to-many transducers lose accuracy for two related reasons. They train against a single EM alignment, and they absorb generative context only as sparse on/off indicator features, which overfit. We claim that replacing the fixed alignment with a posterior-guided latent alignment, and replacing high-order indicator features with a few dense, order-decomposed generative log-probabilities, will reduce word error. The largest gains should appear where alignments are least reliable: small training sets, noisy lexicons, and cross-script transliteration.
The evidence for the gap comes from both literatures. In translation, Phrase Table Training for Precision and Recall shows that one spurious alignment link kills recall for large phrase pairs. Joint Phrase Alignment and Extraction shows that alignment errors cascade irreparably. Consistency-Aware Search for Word Alignment shows that better alignment scores need not mean better translations. The transduction literature has the same structure but has not framed it this way. Joint Processing and Discriminative Training for Letter-to-Phoneme Conversion builds a strong monotone phrasal decoder trained with MIRA (a margin-based online learner), but it assumes the training data are pre-aligned. Letter-Phoneme Alignment: An Exploration patches noisy EM alignments with a hand-designed rule that merges links where the n-best alignments disagree. It also reports that alignment quality measures do not track downstream accuracy, which is exactly the translation paradox reappearing in G2P.
The constraining evidence comes from Integrating Joint n-gram Features into a Discriminative Training Framework. That paper adds joint chunk-sequence histories to the DIRECTL decoder as binary indicators. Accuracy peaks at history length 4 and then declines, because the discriminative learner memorizes long histories that a smoothed generative model would have backed off from. This negative result rules out one design: we will not add higher-order generative structure as more indicators.
Why the Fisher View Matters Here
Exploiting Generative Models in Discriminative Classifiers turns a generative model into features through its gradient $U_X = \nabla_\theta \log P(X\mid\theta)$. It also proves that a classifier on these features asymptotically does no worse than the generative model's own decision rule. The naive transfer of this idea to transduction would be to use the Fisher vector of a joint multigram model as features. We think that naive transfer would fail, and the reason shapes our design.
For a multigram model with chunk-pair probabilities $\theta_k$, the gradient with respect to $\theta_k$ is the posterior expected count of chunk pair $k$ in $(x,y)$ divided by $\theta_k$. So the unlocalized Fisher vector is a reweighted bag of chunk-pair counts, which is nearly the indicator representation that already overfits. In addition, this vector requires the output $y$, which is unknown at test time.
The useful content of the Fisher score is its localized form, the posterior marginal of each chunk pair at each position. This information exists only when $y$ is known, that is, during training. So we use it where it is available: to choose and constrain the alignments the discriminative learner trains on. At test time we use a separate, dense summary of the generative model that can be computed while the output is being built.
Research Program
The first component is a joint multigram aligner. It follows the stochastic-transducer EM of Applying Many-to-Many Alignments and Hidden Markov Models to Letter-to-Phoneme Conversion, with chunk lengths capped as in S-Restricted Monotone Alignments. That paper shows the capped, left-to-right search space for word-length strings is small enough to enumerate exactly. For each training pair, forward-backward gives the posterior $\gamma(i{:}j, k{:}l)$ that letters $i$ through $j$ align to output symbols $k$ through $l$.
The soft extraction set is every chunk pair whose corpus-level posterior mass exceeds a threshold $\tau$. The decoder's inventory of allowed operations is drawn from this set rather than from the Viterbi alignment. This is the transduction analogue of predicting extraction sets directly, as in Discriminative Modeling of Extraction Sets. The difference is that our extraction set comes from a many-to-many generative posterior rather than from a supervised word aligner, and it needs no hand-aligned data. Thresholding the posterior also works as a soft version of the anchor pruning in Inversion Transduction Grammar for Joint Phrasal Translation Modeling. That paper discards any span crossing a high-confidence link. We instead discard spans the posterior considers implausible, so a single bad link cannot remove a whole region of spans.
The second component is the discriminative decoder: a DIRECTL-style monotone phrasal decoder with beam 50 and the standard context-window and transition indicator features. It is trained with latent alignments. For a training word $x$ with gold output $y^*$, the reference alignment at each update is
$$a^* = \arg\max_{a \in \mathcal{A}_\tau(x, y^*)} \; w^\top f(x, y^*, a) + \lambda \log P_{\text{gen}}(a \mid x, y^*),$$
where $\mathcal{A}_\tau(x,y^*)$ is the set of alignments of $x$ to $y^*$ built only from soft-extraction-set chunks, $w$ are the decoder weights, $f$ the decoder features, and $P_{\text{gen}}(a\mid x,y^*)$ the aligner's posterior over alignments. MIRA then moves $w$ toward $(y^*, a^*)$ and away from the decoder's current best guess. The weight $\lambda$ starts large, so early training trusts the generative aligner, and is annealed toward a small value, so the discriminative model can later override alignments the aligner prefers but that hurt prediction. Because the $\arg\max$ ranges over capped monotone segmentations of short strings, it can be computed exactly by dynamic programming.
The third component supplies generative context at test time without indicator explosion. We train a smoothed joint n-gram model over chunk-pair sequences up to order $N=8$ and give the decoder one real-valued feature per order,
$$g_n(y, a) = \log P_n(o_t \mid o_{t-n+1}, \dots, o_{t-1}),$$
summed over positions $t$. Here $o_t$ is the $t$-th chunk-pair operation and $P_n$ is the order-$n$ component of the smoothed model. The learner can then decide how much to trust long histories, using eight weights instead of millions. This follows the logic of NBSVM in Baselines and Bigrams, where generative log-count ratios act as a regularizing prior inside a linear model. We move that idea from classification to structured decoding.
Baselines
The naive reference point is the standard pipeline. It trains DIRECTL on the 1-best EM many-to-many alignment with its default indicator features and no generative scores, as in Joint Processing and Discriminative Training for Letter-to-Phoneme Conversion. That paper reports 90.51% word accuracy on Celex, and we must first reproduce this before any comparison means anything.
The strongest existing method on this question is DIRECTL with joint n-gram indicator features from Integrating Joint n-gram Features into a Discriminative Training Framework. It is the closest published system that combines many-to-many generative structure with discriminative training. That paper reports English-Chinese transliteration word accuracy rising from 73.34% to 74.17% and relative error reductions of 3.6% to 17.3% across letter-to-phoneme sets. We will run it at every history length from 2 to 8 so its best setting is not a straw man.
We also include the best alignment-repair heuristic, EM-Aggregation from Letter-Phoneme Alignment: An Exploration. That paper reports 89.11% word accuracy on Celex-En, and we will feed its merged alignments to the same decoder. As a non-discriminative structured reference we include the CRF-over-restricted-segmentations system from S-Restricted Monotone Alignments, which reports 94.6% on Brulex and 92.6% on German Celex. As a neural reference we include the 4-layer Transformer of Transformer Based G2P. No reported word accuracy for that model on CMUDict or NETtalk was found on Lacuna, so we will train it ourselves.
All systems use identical train, development, and test splits, the same beam of 50 for the phrasal decoders, and the same number of development-set tuning runs, 20 per system over their exposed hyperparameters. Every system also sees the same data: none gets extra lexicons.
Beating the field means a statistically significant word-accuracy gain over the best-tuned indicator system on at least three of the five datasets. The gain must be concentrated in the low-resource and noisy conditions described below, not only in the full-data setting where alignments are already good.
Two ablations isolate the key ingredient. The first keeps the order-decomposed dense features but trains on the fixed 1-best alignment. The second keeps latent alignment but removes the dense features. If the first ablation matches the full system, latent alignment is not doing the work.
Why not just…?
Why not just add the smoothed joint n-gram log-probability as one real-valued feature to DIRECTL? This is the simplest form of generative-discriminative integration, and it is a real threat to the dense-feature half of the proposal. A smoothed model does not overfit at order 8 the way indicators do. What it does not touch is the fixed-alignment problem, because the decoder still learns from one EM alignment per word. Our first ablation, dense generative features with a fixed alignment, is exactly this alternative plus a per-order split. We will also run the single-feature version directly. If it matches the full system, we will report that the benefit comes from smoothing alone and drop the latent-alignment claim.
Why not just train on the union of n-best alignments, or on EM-Aggregation output? Both are cheaper than latent training. EM-Aggregation, however, is reported at 89.11% on Celex-En in Letter-Phoneme Alignment: An Exploration, which is below the 90.51% of the 1-best-trained phrasal decoder in Joint Processing and Discriminative Training, so repairing alignments before training has not clearly helped. An n-best union weights every candidate equally regardless of what the decoder can use. We run both as baselines through the same decoder.
Why not just re-align the training data with the trained decoder and retrain, a hard self-training loop? This is the most serious cheap alternative to our latent training, since it also lets the discriminative model revise alignments. Its weakness is that without the generative posterior term, the decoder can drift toward degenerate alignments that fit its current weights, the same local-optimum problem that plain EM has. We run two to three self-training rounds as a baseline and compare them against the $\lambda=0$ setting of our method. If self-training matches the annealed-$\lambda$ system, the posterior guidance is unnecessary.
Why not just use a Transformer and skip explicit alignment? Neural attention learns soft alignments implicitly. Still, Transformer Based G2P found that deeper models degraded and that lower phoneme error did not reliably mean lower word error. Phonetically Induced Subwords found that explicit phonetic multigram units reduced RNN-T speech recognition error from 10.45% to 9.07% on Librispeech test-clean, so explicit chunk structure still pays off inside neural systems. We expect the Transformer to be competitive at full CMUDict scale and weaker on small subsets. The learning curve below will show whether that holds.
Proposed Experiments
We evaluate word accuracy, and phoneme error rate where the dataset convention reports it, on CMUDict (106,837 train / 12,000 test), Celex-En, NETtalk (14,851 / 4,951), French Brulex, and NEWS English-to-Chinese, Russian and Hindi transliteration. We use each dataset's standard splits and bootstrap significance tests.
Four checks decide whether the idea holds. The first is a learning curve on CMUDict at 2k, 5k, 20k, and full training size. The prediction is that the gap between latent and fixed-alignment training grows as data shrinks, because EM alignments are noisier with less data.
The second is an alignment-corruption test, modeled on the single-bad-link finding in translation. We perturb 5%, 10%, and 20% of 1-best training alignments by shifting one chunk boundary. The pipeline and indicator baselines should degrade roughly in proportion to the corruption, while the latent system should mostly recover the correct alignments, because $a^*$ is re-chosen at every update.
The third is the history-order curve. We plot accuracy against $n$ from 2 to 8 for indicator features and for dense per-order features. The claim fails if the dense curve also peaks at 4.
The fourth is an annealing ablation over $\lambda$: fixed large, fixed zero, and annealed. We also record how often $a^*$ disagrees with the EM Viterbi alignment, and whether those disagreements concentrate on known hard cases such as silent letters and "x" → /k s/. That analysis shows whether the discriminative learner is correcting the aligner or just adding noise.
As a secondary check that the mechanism is not specific to phonetics, we will run the soft-extraction-set pruning, without the decoder changes, on the monolingual phrasal alignment benchmark EDB++P. There the semi-Markov aligner of Semi-Markov Phrase-Based Monolingual Alignment reports 72.8% F1, trained on synthetic merged alignments. The question is whether posterior-derived chunk sets can replace those synthetic merges as training targets.
References
- Phrase Table Training for Precision and Recall: What Makes a Good Phrase and a Good Phrase Pair?
- Joint Phrase Alignment and Extraction for Statistical Machine Translation
- Consistency-Aware Search for Word Alignment
- Joint Processing and Discriminative Training for Letter-to-Phoneme Conversion
- Letter-Phoneme Alignment: An Exploration
- Integrating Joint n-gram Features into a Discriminative Training Framework
- Exploiting Generative Models in Discriminative Classifiers
- Applying Many-to-Many Alignments and Hidden Markov Models to Letter-to-Phoneme Conversion
- S-Restricted Monotone Alignments: Algorithm, Search Space, and Applications
- Discriminative Modeling of Extraction Sets for Machine Translation
- Inversion Transduction Grammar for Joint Phrasal Translation Modeling
- Baselines and Bigrams: Simple, Good Sentiment and Topic Classification
- Transformer Based Grapheme-to-Phoneme Conversion
- Phonetically Induced Subwords for End-to-End Speech Recognition
- Semi-Markov Phrase-Based Monolingual Alignment