Automatic pronunciation tutors judge a learner's speech by asking how well each sound matches a model of the target sound. They ignore the voice source: how tightly the vocal folds close and how strained or breathy the voicing is. Yet acoustic studies of learners show that the voice source itself carries a foreign accent. German speakers phonate with more strained vocal-fold closure than French speakers, and learners shift this behaviour in uneven, proficiency-dependent ways when speaking the other language (Evaluation of Phonatory Behavior of German and French Speakers in Native and Non-Native Speech). We propose to treat phonation the way a particle-physics experiment treats an extra subdetector. It is a separate measurement stream with its own calibrated response model. It is informative only in certain regimes (voiced vowels) and often simply absent. Its evidence is combined with the other streams as a sum of per-stream log-likelihood ratios.
In two sentences: particle identification combines ionization, timing and Cherenkov measurements by adding up per-detector log-likelihood ratios that are calibrated as a function of momentum, and it skips detectors that gave no signal. We build the same structure for L2 speech, with a phonatory stream, a vowel-spectrum stream and a segmental stream calibrated per phone and per native language. We then test whether the phonatory stream adds measurable, diagnosable information beyond self-supervised speech embeddings.
Core Claim
The claim has two parts, and each can be falsified on its own. First, three acoustic measures carry information about non-native pronunciation that existing scorers do not exploit. These are the open quotient gradient (OQG, how long the folds stay open), the glottal opening gradient (GOG, spectral decay tied to closure; smaller means more strained), and incompleteness of closure (IC, derived from the bandwidth of the first resonance of the vocal tract). Second, this information is best used through a context-calibrated, per-stream likelihood ratio rather than by appending raw features to a network. The reason is that the direction of the phonatory "error" depends on the speaker's native language (L1) and is not monotone in proficiency. Pützer et al. found German learners loosen closure toward French norms, while French learners change IC instead. They also found advanced learners of both groups are more strained in their L2 than beginners. A fixed feature weight shared across L1s cannot represent a cue whose sign flips with L1 and whose relation to proficiency is non-monotone. A response model conditioned on L1 can, in the same way that a dE/dx response model must be conditioned on momentum before it means anything.
Mechanism
Each acoustic stream $s$ yields an observation $x_s$ for a segment. For phonation this is the triple (OQG, GOG, IC) measured on the vowel nucleus. For the spectral stream it is the Mel-scaled DCT coefficients plus formant trajectory slices. For the segmental stream it is a phone posterior from a pretrained model. The score contrasting a target-like hypothesis $T$ with a learner-like hypothesis $L$ is
$$\Delta(x) = \sum_{s \in A(x)} w_s(c)\, \ln \frac{P(x_s \mid T, c)}{P(x_s \mid L, c)},$$
where $A(x)$ is the set of streams actually measured for that segment, $c$ is the context (phone identity, voicing, speaker L1), and $w_s(c)$ is a learned weight.
Three choices in this formula come from particle identification. The density models are conditioned on context, as detector responses are conditioned on momentum and pseudorapidity. Streams missing from $A(x)$ drop out of the sum instead of being imputed. The weights $w_s(c)$ correct for the fact that acoustic streams are not conditionally independent; phonation and spectral tilt are correlated, unlike TOF and RICH. The classic fusion that CLEO III used to separate protons, kaons and pions by summing RICH and dE/dx likelihoods (Measurement of the decay rate of $\Xi_c^0 \to p K^- K^- \pi^+$) supplies the template. BABAR's kaon–pion separation from DIRC plus dE/dx (Observation of the Decay $B^+ \to [K^+K^-\pi^+$](/work/observation-of-the-decay-mml-math-xmlns-mml-w9dm)) shows the template working at scale. The per-stream terms are the diagnostic output: a tutor can report that a vowel failed on closure rather than on tongue position.
Relation to Prior Work
Prior work plays three roles here: motivating the gap, supplying methods, and constraining the design.
Pützer et al. motivate the gap by establishing phonatory differences with univariate tests on 81 speakers. They build no predictive model, and their asymmetric, non-monotone findings are what make naive feature-appending suspect.
On the speech side, three scorers define the competition, and all three use phone identities or generic embeddings with no voice-source stream. Kaldi GOP gives a phone-level log posterior ratio, evaluated on speechocean762 (speechocean762). Context-aware GOP reweights phone boundaries by entropy and adds a duration model (Context-Aware Goodness of Pronunciation). A fine-tuned wav2vec 2.0 detector classifies aligned segments (Explore wav2vec 2.0 for Mispronunciation Detection). We keep GOP's likelihood-ratio form and extend it across physical streams.
Two vowel studies supply the spectral stream. Graham et al. show Mel-DCT coefficients track CEFR level (Vowel Characteristics in the Assessment of L2 English Pronunciation). Bakkouche and Post show proficiency appears in when formants move within the vowel (Influence of Proficiency and L2 Experience on Dynamic Spectral Cue Utilization).
One result constrains the design. Graham et al. found no L1 main effect on DCT coefficients. If this generalizes, L1 conditioning may matter for the phonatory stream but not the spectral one, and the model must be allowed to learn that.
From physics we reuse two methods. ALICE's set-based Transformer handles tracks with missing detectors without imputation (Machine-learning-based particle identification with missing data). LHCb's common-specific decomposition separates channel-invariant features from channel-specific ones (Robust Neural Particle Identification Models). We treat both as candidate implementations to be tested, not as assumed wins.
Research Program
First, we build a robust phonatory front end. It estimates OQG, GOG and IC on vowel nuclei and outputs an explicit "not measured" flag when voicing is absent or the first-formant bandwidth estimate is unreliable. We validate it by reproducing the direction of the Pützer et al. contrasts on IFCASL. If our extractor cannot recover the published German–French and beginner–advanced differences, the rest of the program stops there.
Second, we fit context-conditioned density models per stream. These are class-conditional mixture models indexed by phone class and L1, trained on target-language native speech ($T$) and learner speech ($L$). We learn $w_s(c)$ with a small calibration network on held-out speakers.
Third, we compare this explicit fusion against learned fusion. One variant is an ALICE-style set encoder over (stream key, value) pairs. The other is the same encoder with a common-specific head that treats L1 as the "decay channel."
Fourth, we probe frozen wav2vec 2.0 layers for OQG, GOG and IC. This tells us directly whether the embedding already contains the information we are adding.
Baselines
The naive reference is raw phone-level GOP. On speechocean762 it reaches PCC 0.25 with human phone scores, and an SVR on GOP-derived features reaches PCC 0.45 with MSE 0.16 (speechocean762). On L2-ARCTIC a phone-independent GOP gives precision and recall of 0.29 on substitutions (L2-ARCTIC). We rerun the Kaldi recipe on our splits rather than quoting these numbers.
The strongest existing method is the fine-tuned wav2vec 2.0 detector, which reports F1 = 0.610 on L2-ARCTIC (precision 0.580, recall 0.643), above log posterior probability at 0.583 (Explore wav2vec 2.0 for Mispronunciation Detection). It is the strongest competitor because, if phonatory cues matter, a model fine-tuned on raw audio could in principle learn them itself. We reproduce it with the same roughly three-hour fine-tuning subset, identical speaker splits, and the same hyperparameter search budget as our fusion models. "Beating" it means a gain over the reproduced F1 that survives speaker-level bootstrap intervals.
Context-aware GOP reports 67.56% accuracy against 53.44% for GOP, but on a private child dataset. We reimplement its boundary weighting as an extra segmental stream rather than compare numbers.
For IFCASL, no reported result found on Lacuna for automatic proficiency or L1 classification. Our naive reference there is a logistic regression on spectral features alone.
The key ablation removes the phonatory stream from every model, including wav2vec plus fusion. A second ablation removes L1 conditioning from the phonatory density models. If the asymmetry argument is right, this should hurt the phonatory stream far more than the spectral one.
Why not just…?
Why not just fine-tune wav2vec 2.0 and let it learn phonation? It may already encode phonation. Nothing in the file shows it does, and nothing shows it does not. The probing experiment settles this directly. If frozen layers predict OQG, GOG and IC nearly perfectly and adding the phonatory stream gives no gain over the reproduced 0.610 F1, the information claim fails. The remaining contribution would then be the diagnostic decomposition, which the embedding does not provide.
Why not just append OQG, GOG and IC to the feature vector with a missing-value mask? This is the most serious threat. ALICE beat mean and linear imputation, but the file reports no comparison against simple mask indicators. We run masked concatenation into the same SVR and the same wav2vec head as a first-class arm. If it matches the calibrated likelihood fusion, we drop the fusion machinery and report the simpler result. The asymmetry and non-monotonicity in Pützer et al. are our reason to expect it will fall short on held-out L1s. They are a reason, not proof.
Why not just add speaker L1 as an input instead of decomposing or conditioning? For a seen L1 this is nearly equivalent to our conditioning, and we expect parity. The difference appears on an unseen L1, where a one-hot input has no value to take. LHCb's decomposition cut error on unseen decay channels by 97–98% relative to the plain network (Robust Neural Particle Identification Models). That is a different domain, so we test it here with leave-one-L1-out evaluation rather than assuming it transfers.
Proposed Experiments
The first decisive check runs on IFCASL. We classify beginner versus advanced, and L1 versus L2 production, for each speaker group, using spectral streams alone versus spectral plus phonatory. The test is speaker-independent cross-validation over the 81 speakers. We predict the phonatory stream helps most in exactly the cells where Pützer et al. found significant differences, such as German beginners' OQG. We predict it helps little where they did not.
The second check runs on L2-ARCTIC with leave-one-L1-out evaluation across its five L1s. We measure mispronunciation F1, split by vowel versus consonant targets. Phonation is defined only on voiced segments, so we expect any gain to concentrate on vowels and to be near zero on consonant substitutions. A uniform gain across both would suggest leakage rather than a phonatory effect.
The third check runs on speechocean762 and measures PCC with sentence-level accuracy and prosody scores and with phone scores. Its mobile-phone recordings make it a deliberate stress test of whether the phonatory front end survives channel variation. If the phonatory stream helps on studio-quality IFCASL and L2-ARCTIC but hurts on speechocean762, that bounds where this approach is usable.
Across all three, the result we care most about is the pairing of the ablations. The proposal stands if removing phonation costs accuracy on held-out L1s, and if removing L1 conditioning costs more on the phonatory stream than on the spectral stream. Otherwise it should be abandoned.
References
- Evaluation of Phonatory Behavior of German and French Speakers in Native and Non-Native Speech
- Measurement of the decay rate of Ξc0→pK-K-π+ relative to Ξc0→Ξ-π+
- Observation of the Decay<mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML" display="inline"><mml:msup><mml:mi>B</mml:mi><mml:mo>+</mml:mo></mml:msup><mml:mo>→</mml:mo><mml:msup><mml:mi>K</mml:mi><mml:mo>+</mml:mo></mml:msup><mml:msup><mml:mi>K</mml:mi><mml:mo>−</mml:mo></mml:msup><mml:msup><mml:mi>π</mml:mi><mml:mo>+</mml:mo></mml:msup></mml:math>
- speechocean762: An Open-Source Non-Native English Speech Corpus for Pronunciation Assessment
- Context-Aware Goodness of Pronunciation for Computer-Assisted Pronunciation Training
- Explore wav2vec 2.0 for Mispronunciation Detection
- Vowel Characteristics in the Assessment of L2 English Pronunciation
- Influence of Proficiency and L2 Experience on Dynamic Spectral Cue Utilization in L2 Vowel Perception and Production
- Machine-learning-based particle identification with missing data
- Robust Neural Particle Identification Models
- L2-ARCTIC: A Non-native English Speech Corpus