Grapheme-to-phoneme conversion (G2P) turns written text into a pronunciation and sits at the front of every text-to-speech system. In languages where one written character has several pronunciations, text alone often cannot decide which one is meant. A recent Cantonese system therefore gives the model audio as well as text. Because labeled Cantonese recordings are scarce, it trains on speech produced by a commercial text-to-speech engine (Speech-guided Grapheme-to-Phoneme Conversion for Cantonese Text-to-Speech). This setup has a quiet circularity. The synthetic audio was generated from the text by the engine's own pronunciation front-end, so every training pair agrees with one fixed text-to-pronunciation policy. A model trained on it can reach a low error rate by learning that policy from the characters and their context, and can largely ignore the audio. If so, the "speech guidance" will fail on real speakers, which is exactly the setting the paper flags as untested.
We propose a direct test of whether such a model uses the audio, and then a cheap way to predict the real-speech gap without real labels. The tool is adaptive signal decomposition. Empirical mode decomposition (EMD) splits a waveform or a feature trajectory into a few oscillatory components, called intrinsic mode functions (IMFs), ordered from fast to slow, without assuming fixed sine bases. We swap only the components that carry the cue separating two readings of a character, which for most Cantonese polyphones is the tone-bearing pitch and energy contour, and check whether the model's predicted pronunciation follows the swapped audio. The flip rate under these targeted swaps, computed on synthetic audio alone, is our proposed measure of acoustic reliance. Our hypothesis is that it predicts real-speech error better than synthetic held-out error does.
Core Claim
A speech-guided G2P model trained only on synthetic speech mostly imitates the synthesizer's text front-end. Its measured gain over dictionary tools comes mainly from text context, not from acoustics. Counterfactuals built by substituting tone-bearing IMFs expose this. The resulting flip rate tracks how much real annotated speech the model needs before it truly listens, which turns the substitutability question into a diagnostic that needs no real labels. A substitutability curve asks how much real data a synthetic-pretrained model needs; On the Equivalency, Substitutability, and Flexibility of Synthetic Data measures such curves for video tracking.
Positioning
Three strands motivate the work. The Cantonese speech-guided paper supplies the system under study. It fine-tunes an intermediate layer of an off-the-shelf speech recognizer on Azure-synthesized speech oversampled for 88 polyphonic characters. Its reported jyutping error rate (JER, the fraction of syllables whose romanized Cantonese pronunciation is wrong) is 2.0% on the MDCC test set, against 6.8% for ToJyutping and 4.6% for PyCantonese, and 1.5% on polyphones against 10.6% for ToJyutping. Those baselines are dictionary and rule tools. None of them is a learned text model trained on the same engine-derived labels, so the comparison cannot separate "audio helps" from "a neural model trained on Azure's choices helps."
Bridging the Stability-Expressivity Gap supplies the second piece of motivation. In Thai and Lao spoken language models, scaling synthetic speech flattens prosody, which the authors call synthetic erosion. They measure it with token-level prosodic entropy and listener ratings. Pitch contour is precisely the cue a tonal G2P model would need, so erosion predicts that synthetic tone cues are both cleaner and less variable than real ones. We test that prediction at the level of physical signal components, not discrete tokens, and inside a G2P model, not a model that bypasses G2P.
The third strand is negative. Data augmentation for low-resource grapheme-to-phoneme mapping found that synthetic substring augmentation hurt G2P across ten languages. That result warns that synthetic data in G2P can look helpful while teaching the wrong thing.
The method comes from two places. IMFACT builds counterfactual time series by swapping individual IMFs with those of a nearest example from another class. It reports 100% validity and high range validity on non-speech benchmarks. We reuse its substitution logic but apply it to speech, with a known target class (the alternative reading) instead of a searched neighbor. Complete ensemble EMD with adaptive noise (CEEMDAN), as used in Friction Signal Denoising Using Complete Ensemble EMD with Adaptive Noise and Mutual Information, supplies the decomposition. It adds noise at each stage so components separate consistently and reconstruct exactly, and it uses mutual information between successive IMFs to locate the boundary between noise components and signal components. Noise-robust speech feature processing with empirical mode decomposition shows that EMD of frame-level log-energy trajectories isolates meaningful speech dynamics better than RASTA filtering, a standard fixed temporal filter for speech features. We follow its choice to decompose feature trajectories as well as raw waveforms.
What prior work does not do is our contribution: none of it asks whether a synthetic-trained pronunciation model uses its acoustic input, and none uses component-level counterfactuals as a label-free predictor of the real-world gap.
Research Program
Training conditions. We first reproduce the speech-guided recipe on Cantonese, using Azure synthesis over a polyphone-oversampled corpus. We then train the same architecture with varying amounts of real annotated speech added during fine-tuning: 0, 5, 10, 25, 50, and 100 percent of a real pool. The real pool is drawn from MDCC and from the Cantonese portion of Common Voice, with polyphonic syllables hand-verified by two annotators. This yields a substitutability curve in the sense of Chang et al.
Counterfactual construction. For each polyphonic character in a synthetic test sentence, we synthesize a second rendering in which that character is replaced by a homophone carrying the alternative reading. This gives two recordings that differ only in the target syllable. We align them at the syllable level, decompose both target segments with CEEMDAN, and use the mutual-information criterion to separate fast noise-like components from slower structure.
We then build two kinds of counterfactual. The waveform-level version swaps the IMF bands that contain the fundamental frequency and low harmonics. The trajectory-level version decomposes the F0 and log-energy tracks themselves, swaps their slow IMFs, and resynthesizes with a source-filter vocoder.
Validity check. A counterfactual counts as valid only if an independent Cantonese tone classifier, not the model under test, labels it with the alternative tone. We report this validity rate, as IMFACT does.
Acoustic reliance score. The score is the fraction of valid counterfactuals on which the G2P model's output moves to the alternative reading, while the written text still shows the original character. We contrast it with control swaps of fast or mismatched IMFs, which should not flip the prediction.
Multilingual extension. To keep the claim from being a Cantonese curiosity, we repeat the probe on Thai. Thai is tonal, and its orthography leaves vowel length and syllable-boundary ambiguity that audio can resolve. Real audio comes from Common Voice, the same source Geng et al. use. We found no reported G2P result on Lacuna for Thai speech-guided G2P, so Thai results are exploratory.
Baselines
The naive reference point is a text-only learned G2P model. It is a Transformer encoder-decoder over characters with sentence context, trained on exactly the same sentences and the same engine-derived pronunciations, with no audio. It gets the same tuning budget (an identical hyperparameter search, measured in GPU-hours) and the same training steps as the speech-guided model. This is the most important comparison in the proposal, because the published comparison only covered dictionary tools. If this model approaches the speech-guided model's 2.0% MDCC JER and 1.5% polyphone JER, the audio is contributing little.
The strongest existing method is the speech-guided system itself, reproduced from the description in Speech-guided Grapheme-to-Phoneme Conversion for Cantonese Text-to-Speech. That includes the sub-syllable decomposition into initial, middle, and final, the dictionary-guided embeddings, and the inference-time masking. Its reported numbers are 2.0% JER on MDCC and 1.5% on polyphones. If our reproduction lands more than one point away on MDCC, we report it against our own reproduction and note the gap. The dictionary tools ToJyutping (6.8% JER) and PyCantonese (4.6% JER) appear as floor references only.
For the probe, the key ablation compares the targeted slow-mode swap with three alternatives: a swap of fast IMFs, a random IMF swap, and a fixed Fourier band swap over the same frequency range. All four use identical alignment and resynthesis. The targeted swap must beat the controls in flip rate at matched validity. Otherwise the decomposition is not doing the work.
For the predictive claim, the comparison is between two predictors of real-speech JER across training checkpoints and real-data fractions: synthetic held-out JER, the natural indicator, and our reliance score. Our score counts as better if its rank correlation with real JER exceeds that of synthetic JER by a margin outside the bootstrap interval.
Why not just…?
Why not just measure the substitutability curve on real speech and skip the decomposition? The curve tells you how much real data closes the gap, but not why the gap exists, and it needs real labels at every point. Our probe uses only synthetic audio. The curve is still part of the program because it is the ground truth the probe must predict. If synthetic held-out JER already tracks real JER well, the probe adds little, and the correlation comparison above would show that.
Why not just edit pitch directly with a vocoder instead of swapping IMFs? This is a real threat. For polyphones that differ only in tone, direct F0 editing is a simpler and cleaner counterfactual. We include it as a separate probe arm. EMD earns its place only if it does better on polyphones whose readings also differ segmentally, in vowel quality or in syllable-final stops, where swapping energy and spectral-band modes can carry cues that a pitch edit cannot. If F0 editing matches EMD on every polyphone class, we will report that pitch editing suffices and narrow the claim to the reliance finding itself.
Why not just diversify the synthetic data with more voices, more engines, and added noise? Diversity may reduce acoustic overfitting. But every engine reads text with its own front-end, so the audio still mirrors a text policy, and the shortcut through text survives. We test this by adding a multi-voice, noise-augmented training arm and measuring its reliance score. If diversity alone raises flip rates and closes the real-speech gap, that becomes a simpler recommendation, and we will say so.
Why not just use a stronger text model with wider context? If a context-rich text model matches the speech-guided system on real polyphones, audio guidance is unnecessary for Cantonese text-to-speech. That would be a useful negative result, and the text-only baseline is designed to detect it. The cases where text truly cannot decide, such as the same sentence read two ways, are covered by a small hand-recorded set in which speakers deliberately produce both readings.
Decisive Checks
Three outcomes would settle the proposal.
The first is whether the speech-guided model trained on synthetic data shows a near-zero flip rate under valid targeted swaps while the real-fine-tuned versions flip substantially. That outcome would confirm that synthetic training teaches the text policy, not listening.
The second is whether the flip rate rises monotonically along the real-data curve and predicts real-speech JER, especially on noisy Common Voice audio, better than synthetic JER does. If the flip rate is high even for the synthetic-only model, our circularity hypothesis is wrong. The finding would then be that synthetic speech does teach acoustic disambiguation, which would strengthen the case for synthetic reliance in G2P.
The third is whether the targeted-swap advantage over Fourier-band and random swaps holds in Thai as well as Cantonese, which would show the probe is not tied to one language.
References
- Speech-guided Grapheme-to-Phoneme Conversion for Cantonese Text-to-Speech
- On the Equivalency, Substitutability, and Flexibility of Synthetic Data
- Bridging the Stability-Expressivity Gap: Synthetic Data Scaling and Preference Alignment for Low-Resource Spoken Language Models
- Data augmentation for low-resource grapheme-to-phoneme mapping
- IMFACT: Counterfactual Explanations for Time Series via Intrinsic Mode Function Substitution
- Friction Signal Denoising Using Complete Ensemble EMD with Adaptive Noise and Mutual Information
- Noise-robust speech feature processing with empirical mode decomposition