Grapheme-to-phoneme conversion, or G2P, turns a written word into its sequence of sounds. Text-to-speech and speech recognition both depend on it for every word missing from the pronunciation dictionary. Modern neural G2P models write the pronunciation one sound at a time, and nothing stops them from producing outputs no speaker would accept: two primary stresses in one word, a syllable with no vowel, a doubled or dropped consonant. Older systems showed that banning such outputs pays off. Forcing one vowel per syllable and one main stress per word cut German word error from 21.5% to 13.7% (Phonological Constraints and Morphological Preprocessing for Grapheme-to-Phoneme Conversion). In that system, however, the rules were bolted onto the search procedure of a count-based model, and the model never learned with them in place.
We propose a G2P model whose training objective already knows these rules. A bidirectional recurrent encoder reads the whole spelling. A conditional random field (CRF), a model that scores entire output sequences jointly rather than one symbol at a time, then sums over every way of cutting the spelling into chunks and mapping each chunk to a short run of sounds. Only paths that obey the word-level rules are included in that sum. The question is whether learning inside the legal output space beats the two common habits: learning freely and filtering at decoding time, or adding a sound-sequence language model afterwards. The answer should matter most for low-resource languages, where a few hundred to a few thousand dictionary entries are not enough to learn the rules implicitly.
Core Claim
A globally normalized, constraint-aware model should beat the same model trained without constraints and pruned only at decoding time, especially when training data are scarce. "Globally normalized" means the probability of a pronunciation is computed by normalizing over all complete legal outputs, not one step at a time. The model has three parts. A BiLSTM (a bidirectional long short-term memory network, which reads the spelling left-to-right and right-to-left) scores grapheme spans. A semi-Markov CRF (a CRF whose units are variable-length segments) treats the grapheme-to-phoneme alignment as a hidden variable. Finally, the CRF's states are augmented with small counters that track phonological well-formedness.
The reason training-time constraints should help is that the encoder stops spending capacity on probability mass that decoding would discard anyway. The training signal then concentrates on genuinely ambiguous choices, for example which syllable carries stress, rather than on whether some syllable does. A pure decoding-time constraint cannot provide this.
Model
Let $x = x_1 \dots x_T$ be the graphemes. A segmentation $s$ cuts $x$ into spans of length at most $K$, with $K = 3$ covering cases like "tch". Each span is labeled with a phoneme chunk $c$ of zero to three phonemes. The empty chunk covers silent letters, and three phonemes cover cases like "x" → /k s/. Concatenating the chunks gives the output $y$. Each step also carries a small automaton state $q$ that records two things: whether a primary stress has already been emitted, and whether the current syllable already has a nucleus. The model is
$$p(y \mid x) = \frac{1}{Z(x)} \sum_{s,\,q \,\in\, \mathcal{A}(x,y)} \exp\Big(\sum_{j} \phi_{\theta}(c_j, x_{a_j:b_j}) + \psi(c_{j-1}, c_j) \Big).$$
Here $\mathcal{A}(x,y)$ is the set of segmentations and automaton paths that yield $y$ and end in an accepting state, meaning exactly one primary stress and a nucleus in every syllable. The term $\phi_\theta$ is a span score computed from the BiLSTM states at the span edges $a_j, b_j$. The term $\psi$ is a learned transition score between consecutive chunks, which captures local phonotactics. $Z(x)$ sums the same quantity over all legal outputs.
Both sums are computed exactly by a forward pass over a lattice with on the order of $T \cdot K \cdot |C| \cdot |Q|$ entries, where $C$ is the set of candidate chunks and $|Q| \le 4$ is the number of automaton states. Decoding uses Viterbi search over the same lattice, so every output is legal by construction.
The size of the chunk inventory is the practical bottleneck. We limit it by borrowing the local-filtering idea of Filtered Semi-Markov CRF. That paper prunes candidate spans for named-entity recognition with a cheap span classifier before global search, and trains the filter and the CRF jointly. Here the filter keeps the top-$k$ chunks per grapheme span. The difference from that paper is that our spans map into a separate output vocabulary of variable length, not onto entity labels.
Relation to Prior Work
Two CRF systems motivate the design and supply its skeleton.
Grapheme-to-phoneme conversion using conditional random fields trains a linear-chain CRF on HMM-pre-aligned data with hand-built n-gram features. It reports 35.7% WER on NETtalk. Unlike that work, we do not fix the alignment in advance.
Hidden conditional random fields with M-to-N alignments treats the alignment as hidden inside a finite-state CRF and reaches 11.8% WER on English CELEX. We keep that hidden-alignment idea. We replace its discrete windowed feature tables with a learned bidirectional encoder, and we add constraint states to the lattice. That paper also warns that this objective is non-convex and needed IBM Model 1 initialization. We take the warning seriously and include an initialization ablation.
Neural sequence-labeling work shows both what helps and where it stops. Language-Agnostic Syllabification with Neural Sequence Labeling finds that a CRF layer on a BiLSTM beats a per-token softmax for syllable boundaries. Its input is already phonemes, so it never faces the length mismatch between spelling and sound. Sequence Labelling for G2P handles that mismatch with Gram-CTC on a BiLSTM. CTC (connectionist temporal classification) is a loss that sums over alignments but treats output symbols as independent given the alignment, so it cannot learn which phoneme follows which.
Attention-based encoder-decoders such as Jointly Learning to Align and Convert Graphemes to Phonemes reach 20.24% WER on CMUDict with an ensemble. That paper also documents coverage failures such as "KITTIWAKE" → [K IH T AH W W K K]. A monotone segment lattice cannot produce such an output.
Predicting Pronunciations with Syllabification and Stress with RNNs enforces the one-primary-stress rule with a finite-state transducer after an LSTM has already generated its output. It is the closest neural precedent for constraint handling, and our central ablation separates exactly its approach (constraints applied after training) from ours (constraints present during training).
Research Program
We will build the filtered semi-Markov CRF with constraint-augmented states on a two-layer BiLSTM encoder over characters. Korean will use jamo decomposition, which SIGMORPHON 2020 showed is essential.
The constraint automaton is deliberately minimal and language-general. It has a primary-stress counter and a nucleus flag, active only for languages whose transcriptions mark stress or syllables. Nucleus membership comes from the IPA vowel class. Phonotactics beyond these two rules are left to the learned transition term $\psi$, so the comparison is not stacked with hand-written rules.
Three analyses accompany the accuracy results. First, we measure how often unconstrained baselines actually produce illegal outputs, since the whole approach is only worth its cost if this rate is non-trivial. Second, we break down errors into stress, segment-count, and substitution errors. Third, we swap the BiLSTM for a Transformer encoder under the identical CRF head. This tests the observation that LSTMs beat Transformers at about 4.5k training words (SIGMORPHON 2020) while Transformers win on the 106k-word CMUDict (Transformer Based G2P).
Baselines
The naive reference point is the pair n-gram model, the finite-state baseline distributed with SIGMORPHON 2020. For English CELEX, the hidden CRF without M-to-N alignment and transition penalties, which reports 76.8% WER (Hidden CRFs with M-to-N alignments), shows how badly an unstructured lattice does.
The strongest neural competitors are of three kinds. The first is the official SIGMORPHON 2020 BiLSTM-attention encoder-decoder. The second is the Transformer, including the multi-task G2P/P2G ensemble of Frustratingly Easy Multilingual G2P, which reports 14.43 WER on the dev set. The third is the imitation-learning transducer of CLUZH, which reports 14.13% WER and 2.82% PER averaged over the 15 test languages with an 11-model ensemble. CLUZH is the strongest because it already makes monotone edit decisions and so avoids the coverage errors of attention.
Ensembles muddy comparisons. We will therefore retrain single-model versions of the seq2seq and Transformer baselines, and of CLUZH if its code can be run. Each gets the same 3.6k training words, the same dev set for early stopping, and the same hyperparameter search budget of 30 trials. Ensembles of equal size are compared only against each other.
On the classic lexicons, the bar is 11.8% WER on English CELEX (Lehnen et al.) and 13.7% WER on German CELEX (Demberg et al.). On SIGMORPHON 2022 the strongest structured baseline is Gram-CTC on a BiLSTM (Sequence Labelling for G2P). No reported per-language number was found on Lacuna for it, so we will rerun it.
The key ablations hold the encoder fixed and vary two things. One is the CRF trained without constraint states, decoded with or without them. The other is a locally normalized version of the same segment model, which isolates the effect of global normalization from the effect of the lattice.
Beating the strongest single-model baseline means lower average WER over five seeds, with a paired bootstrap test across test words, on the SIGMORPHON 2020 average and on at least the 1k and 100-word tracks of 2022. Matching the 13.7% German CELEX figure without Demberg's morphological preprocessing would also count.
Why not just…?
Why not just decode an ordinary seq2seq model with a constraint transducer, as earlier work did? This is the main threat, and the existing evidence favors it in part. The German gain from 21.5% to 13.7% came entirely from decoding-time pruning, and Predicting Pronunciations with Syllabification and Stress with RNNs did the same for neural models. An autoregressive model's beam, though, may contain no legal hypothesis, or only poor ones, because the model was never trained to rank legal alternatives. Our ablation trains the identical CRF without constraints and decodes it with them, and we also run constrained beam search on the seq2seq baseline. If either matches the constraint-trained model, the central claim fails and we will report it.
Why not just add a phoneme n-gram language model to Gram-CTC? Rescoring would supply local phonotactics. It cannot enforce word-level counts such as exactly one primary stress, and its scores are fused after training instead of shaping the encoder. We will run Gram-CTC with shallow fusion of a phoneme 3-gram model as an explicit baseline. If it closes most of the gap, the transition term $\psi$ is not earning its place, and the contribution narrows to the stress and nucleus states.
Why not just ensemble Transformers or use a pretrained encoder? Ensembles help: 1.50 WER in Frustratingly Easy Multilingual G2P and a 16% relative reduction for CLUZH. These gains are orthogonal to ours, so we compare equal-size ensembles. Pretraining, as in GBERT or T5G2P, needs large unlabeled corpora that many SIGMORPHON languages lack. It also gives no guarantee that outputs are legal. A GBERT encoder could replace the BiLSTM under our CRF head, and we include that variant on the four SIGMORPHON 2021 languages it was built for.
Proposed Experiments
The main experiments run on SIGMORPHON 2020 (15 languages, 3.6k training words each), on SIGMORPHON 2022 (1k, 100, and transfer tracks), on English and German CELEX, and on CMUDict to check behavior at large scale. Metrics are WER and PER, plus two diagnostics: the rate of illegal outputs and the stress error rate wherever stress is marked.
Three outcomes would settle the question. The first is the illegal-output rate of unconstrained baselines. If this is near zero even at 100 training words, constraints cannot help, and the project reduces to a neural hidden-alignment CRF compared against CLUZH. The second is the gap between training with constraints and only decoding with them, as a function of training size. We expect it to shrink toward zero on CMUDict and to be largest on the 100-word track. The third is whether the hidden-alignment CRF, free of attention's coverage failures, removes the over- and under-generation errors that make up a measurable share of seq2seq errors on CMUDict. We count these directly by comparing output length against the gold pronunciation.
We will also test sensitivity to initialization, starting from random weights versus from an EM aligner, because the hidden-alignment objective is known to be non-convex.
References
- Phonological Constraints and Morphological Preprocessing for Grapheme-to-Phoneme Conversion
- Filtered Semi-Markov CRF
- Grapheme-to-phoneme conversion using conditional random fields
- Hidden conditional random fields with M-to-N alignments for grapheme-to-phoneme conversion
- Language-Agnostic Syllabification with Neural Sequence Labeling
- SIGMORPHON 2022 Shared Task on Grapheme-to-Phoneme Conversion Submission Description: Sequence Labelling for G2P
- Jointly learning to align and convert graphemes to phonemes with neural attention models
- Predicting Pronunciations with Syllabification and Stress with Recurrent Neural Networks
- The SIGMORPHON 2020 Shared Task on Multilingual Grapheme-to-Phoneme Conversion
- Transformer Based Grapheme-to-Phoneme Conversion
- Frustratingly Easy Multilingual Grapheme-to-Phoneme Conversion
- CLUZH at SIGMORPHON 2020 Shared Task on Multilingual Grapheme-to-Phoneme Conversion
- Neural Grapheme-To-Phoneme Conversion with Pre-Trained Grapheme Models
- T5G2P: Using Text-to-Text Transfer Transformer for Grapheme-to-Phoneme Conversion