Most automatic pronunciation checkers can only say that a learner swapped one sound for another. They cannot flag an extra sound, as when a Spanish speaker says "esport" for "sport", or a dropped one, as when a final consonant disappears. They also work only for the language pair their error lists were written for. This proposal targets both gaps with one observation from phonology: most added and dropped sounds are not random. They occur where the target word contains a sound pattern the learner's first language forbids, such as a word-initial /s/+consonant cluster or a consonant at the end of a syllable. That pattern can be read off the phonotactics of the two languages, meaning the rules for which sound sequences are allowed. Doing so needs no recordings of learners at all.
We propose to build, for any pairing of target language and first language, a small graph of plausible learner errors around each prompt. Substitution branches come from how articulatorily similar two sounds are, and insertion and deletion branches come from phonotactic conflicts between the two languages. We then score the learner's audio against this graph with a multilingual acoustic model that was never trained on the target language's learners. The question is whether insertion and deletion detection can transfer zero-shot, meaning with no labeled examples, to new target languages. So far this has never been measured; every alignment-free detector has been tested only on English.
Core Claim
Standard scoring aligns each slice of audio to one sound in the prompt and computes Goodness of Pronunciation (GOP), a likelihood ratio of the intended sound against alternatives. William et al. and Cao et al. document that this slot-by-slot alignment hides insertions and deletions. Cao et al. fix this with alignment-free scoring built on Connectionist Temporal Classification (CTC), a training objective that lets a model emit or skip symbols without a fixed frame-to-sound alignment. Their GOP-CTC-AF-SDI variant compares the canonical sequence against every sequence reachable by substitutions, deletions and insertions. Parikh et al. show that restricting that space with hand-built confusion maps cuts computation by roughly 90% and improves precision (Matthews correlation 0.395 on Speechocean762, 0.595 on MPC). The restriction, however, is written by hand for one language pair.
Our claim is that the restriction can be computed rather than hand-written, and that computing it from phonotactic conflict is what makes insertion and deletion detection portable across target languages. Concretely, for a canonical sequence $c$ and learner audio $x$, the score for position $i$ is
$$\mathrm{GOP}(i) = \log P(c \mid x) - \log \sum_{\pi \in G_i(T, L)} P(\pi \mid x),$$
Here $P(\cdot \mid x)$ is the CTC sequence probability from a multilingual acoustic model, and $G_i(T,L)$ is the set of sequences that differ from $c$ at position $i$ only by arcs licensed for target language $T$ and first language $L$. The best-scoring path through $G_i$ doubles as the diagnosis: it says which kind of error occurred and what was produced instead. What is new is not alignment-free scoring itself, and not articulatory distance, which Xu et al. already use to map unseen phonemes. It is building $G_i$ for all three error types from language-level knowledge alone, and testing whether the result holds on target languages the system has never scored.
Research Program
The acoustic side draws on two lines of work that point in different directions. Xu et al. find that multilingual pretraining (XLSR-53) is what makes zero-shot phoneme recognition work on unseen languages. They report 63.6% phonetic token error on BABEL Cantonese and 22.9% average phoneme error on Multilingual LibriSpeech. Yen et al. reach a further point: predicting language-universal articulatory attributes first cuts zero-shot Japanese syllable error from 64.92% (phoneme-based) to 40.08%. In the other direction, the joint assessment/MDD study finds XLS-R noticeably worse than monolingual English backbones for English mispronunciation detection. This constrains our design. We will not assume a multilingual backbone helps; we will test it.
Our acoustic model follows Phonological-Level Mispronunciation Detection and Diagnosis, which predicts 35 binary phonological features with a multi-label CTC head. Trained only on native English, it transfers to accented English with false acceptance under 30% for most features, against 63% for phoneme models. We train the same head on native read speech from Multilingual LibriSpeech languages, with the target language held out. We then convert feature posteriors into phone posteriors for any target inventory, so that $P(\pi \mid x)$ is defined even for phones never seen in training. The comparison between an XLS-R and a WavLM backbone under this articulatory head is our direct test of the tension raised by the joint MDD study.
The error-graph side has three parts. Substitution arcs connect each canonical phone to target-inventory phones within a small Hamming distance over articulatory features, reusing the mapping of Xu et al.. Deletion arcs are placed on consonants whose syllable position (onset cluster member, coda, cluster-final) has low probability under $L$'s phonotactics. Insertion arcs add $L$'s default epenthetic vowel, or a neutral central vowel when that is unknown, at boundaries where the phone bigram or cluster is illegal in $L$. The phonotactic probabilities come from a pronunciation lexicon or grapheme-to-phoneme transcribed text of $L$. None of this touches learner speech.
The same construction therefore serves Korean learners of French and Spanish learners of English without anyone writing rules. That is the portability that the hand-built confusion maps of Parikh et al., the Gujarati interference rules of the vowel detection study, and the expert error lists that Lee & Glass found covered only 49.5% of real patterns all lack.
Detection thresholds are the part most likely to break under transfer, so we treat calibration as its own study. Thresholds are tuned per error type on English L2-ARCTIC and applied unchanged to other target languages. We then separately measure how much a handful of target-language native utterances recovers. This uses native speech only, never learner labels.
Baselines
The naive reference point is forced-alignment GOP with a threshold. Following the Kaldi recipe released with speechocean762, where raw GOP reaches only 0.25 correlation with human phone scores, we run it on the same multilingual acoustic model so that only the scoring method differs. By construction it cannot report insertions. We therefore score its insertion recall as zero and compare it on substitutions and deletions only.
The strongest existing method for all three error types is unrestricted alignment-free scoring, GOP-CTC-AF-SDI from Cao et al.. We run it on the same acoustic model with the same threshold-tuning data, so that the one difference is the restriction set $G_i$. The strongest restricted method is PP-AF GOP with hand-built confusion maps from Parikh et al.. It reports MCC 0.395 on Speechocean762, but that is an English, in-language figure. It can only be run where confusion maps exist, so we run it on English and treat it as an upper reference for what hand curation buys.
For the multilingual Korean-learner corpus we found no reported mispronunciation-detection result on Lacuna. There, the comparison is internal to our runs. On L2-ARCTIC, the supervised wav2vec 2.0 detector of Xu et al. reports F1 0.610 with about 3 hours of learner labels, which marks the supervised ceiling we do not expect to match.
Beating the baselines means two things. First, on held-out target languages our system must reach higher MCC than unrestricted SDI and a higher insertion-and-deletion F1 at a matched false-alarm rate. Second, it must use a decoding graph at least several times smaller. If our system only equals unrestricted SDI in accuracy while being cheaper, we will report that as a speed result, not an accuracy one.
Two ablations isolate the key ingredient. The first replaces the phonotactic deletion and insertion arcs with the same number of arcs placed at random positions. The second builds the graph from the wrong first language, for example Spanish phonotactics for Korean learners. If the wrong-L1 graph performs as well as the right one, phonotactic conflict is not what drives the gains.
Why not just…?
Why not just allow every insertion and deletion and let the acoustic model decide? This is the most serious competitor. With a strong acoustic model, unrestricted GOP-CTC-AF-SDI might match us. The evidence against it is indirect. On real child errors, Cao et al. found the substitution-plus-deletion variant, not the full SDI one, performed best, suggesting that unrestricted insertion arcs add false alarms. Parikh et al. found restriction improves precision in English. Neither result is cross-lingual, so the matched-false-alarm comparison on held-out languages settles it. A tie there turns our claim into one about computation cost alone.
Why not just run a multilingual phone recognizer and compare its output to the prompt by edit distance? Edit distance yields substitutions, deletions and insertions for free. But free recognition on an unseen language has very high raw error: 63.6% on Cantonese in Xu et al.. Every recognition error then becomes a false mispronunciation flag, and the phonological MDD study shows phoneme-recognition detectors accept 63% of errors as correct even within English. We include this as a baseline on the same acoustic model, so the claim that a prompt-anchored graph beats free recognition is measured rather than assumed.
Why not just fine-tune on learner data in one language and apply it elsewhere? The Korean-learner corpus has 400 hours of English, so a reviewer might train a supervised detector on English and apply it to French or Japanese. We run this as a transfer baseline. Its weakness is that learner errors depend on the target language's phonotactics, not only on the learner. Korean learners do not insert the same vowels in French as in English. A detector trained on English error locations has no mechanism for new sites, and per-target-language insertion recall will show whether this is true.
Why not just prompt a speech-capable large model? Wang et al. report GPT-4o at 0.241 phone-level correlation against 0.693 for a dedicated model, with 41–48% of responses unscorable. The masked-token method of Liu et al. is truly zero-shot, but it only produces a sentence-level score (0.60 correlation) and cannot locate an error. Neither gives the position-level, typed diagnosis we target, so we do not rerun them beyond citing these figures.
Proposed Experiments
The main test uses the Korean Learners' L2 Speech Corpus of Seven Languages. It holds the learner's first language fixed and varies the target across Chinese, Japanese, French, German, Spanish and Russian, which is exactly the cross-lingual shift we care about. Its substitution, omission and insertion tags come from automatic forced alignment and have moderate rater agreement (Krippendorff's alpha around .65). We will therefore have trained phoneticians relabel a stratified sample of several hundred utterances per language, and report results on that verified subset. The full automatic tags are used only as a secondary check.
English L2-ARCTIC, which has manual labels for 5,199 substitutions, 1,048 deletions and 497 additions across Arabic, Hindi, Korean, Mandarin and Spanish speakers, serves as the tuning language. It also supports a held-out-L1 test within English, and Speechocean762 provides a cross-check against published English numbers. For each target language we exclude its native speech from acoustic training. A second, easier setting that allows target native speech but no learner speech is reported separately.
We report precision, recall and F1 separately for substitutions, deletions and insertions. We also report MCC for comparison with Parikh et al., false acceptance and diagnostic error rates for comparison with the phonological MDD study, and graph size and decoding time.
Following the concern raised by Cao et al., insertions between prompt phones are scored against an expanded reference that has a slot at every boundary. Otherwise standard metrics would silently ignore them.
Three results would be decisive. First, insertion and deletion F1 on unseen target languages must exceed unrestricted SDI and edit-distance recognition at matched false-alarm rates. Second, the wrong-L1 and random-arc ablations must lose most of that gain; if they do not, the phonotactic idea is refuted even if the system works. Third, the XLS-R versus WavLM comparison under the articulatory head will show whether the multilingual-backbone penalty seen in English MDD disappears once scoring goes through universal features. It will also show whether that penalty, as we expect, reverses on held-out target languages.
References
- Using human perception for automatic accent assessment
- A Framework for Phoneme-Level Pronunciation Assessment Using CTC
- Enhancing GOP in CTC-Based Mispronunciation Detection with Phonological Knowledge
- Simple and Effective Zero-shot Cross-lingual Phoneme Recognition
- A Bottom-up Framework with Language-universal Speech Attribute Modeling for Syllable-based ASR
- A Joint Model for Pronunciation Assessment and Mispronunciation Detection and Diagnosis with Multi-task Learning
- Phonological-Level Mispronunciation Detection and Diagnosis
- Vowel mispronunciation detection using DNN acoustic models with cross-lingual training
- Mispronunciation detection without nonnative training data
- speechocean762: An Open-Source Non-Native English Speech Corpus for Pronunciation Assessment
- Explore wav2vec 2.0 for Mispronunciation Detection
- Exploring the Potential of Large Multimodal Models as Effective Alternatives for Pronunciation Assessment
- Zero-Shot Automatic Pronunciation Assessment
- Constructing Korean Learners’ L2 Speech Corpus of Seven Languages for Automatic Pronunciation Assessment