Telling Errors Apart from Accents and Emotion in Articulatory Pronunciation Feedback

Pronunciation tutors, the software that listens to a language learner and points out mistakes, are moving from giving a single score ("72/100") to giving physical instructions ("raise your tongue, round your lips"). These systems all compare the learner against one neutral, standard target pronunciation. Real speakers depart from that target for three different reasons. They may have made a genuine error. They may be using a legitimate regional variant. Or they may be anxious, excited, or frustrated, and that emotional state physically changes how the tongue and lips move. A tutor that cannot tell these apart will tell a Levantine or Sichuanese speaker to "fix" their accent, and will tell a nervous learner to move their tongue when the real cause is arousal.

We propose a tutor that works in articulatory space, meaning estimated tongue and lip positions recovered from audio. It treats a learner's movement as acceptable if it falls inside the range expected for any accepted regional variant of the target sound, after correcting for the speaker's current emotional arousal. The arousal estimate is continuous, not a category label. Feedback is given only on what remains after that correction, and it is phrased as the direction from where the learner's articulators are to the nearest acceptable target. The affective-granularity claim is narrow and testable. We expect the distortion that emotion causes in articulation to scale with emotional intensity and arousal within a single emotion category, so a categorical emotion label is too coarse to correct for it.

Core Claim

Current mispronunciation detectors produce two kinds of false alarm that no existing system separates from true errors. The first comes from dialect, the second from affect. Both can be suppressed without losing sensitivity to real errors by scoring each phone against a target distribution in articulatory space that is conditioned on the accepted variant and on continuous arousal.

The motivation comes from three findings. In An Exploratory Study of the Relations Between Perceived Emotion Strength and Articulatory Kinematics, electromagnetic articulography (sensors glued to the tongue and lips) showed that tongue-tip position and articulator velocity shift with perceived emotion strength, enough to separate weak from strong emotion at 63.8% accuracy. In Speech Emotion Recognition with Phonation Excitation Information and Articulatory Kinematics, tongue and lip kinematics changed enough across emotions to raise emotion recognition from 79.97% to 88.42%. On the dialect side, speechocean762 handles regional variation administratively: annotators vote on one canonical phone sequence per utterance before scoring. That is evidence the problem exists, not a model of it.

The prior work that comes closest to our goal does not cover either source of false alarm. Articulatory feedback systems instruct learners toward one standard target with no notion of variant or affect. Visualization for improving foreign language pronunciation maps the first two vowel resonances onto a vowel chart calibrated to each speaker. Dhvani gives tongue and lip instructions for Hindi. ViSTAFAE displays estimated vocal-tract shapes. The emotion-and-articulation papers above are analytical only and give no feedback. Bimodal Emotion Recognition based on Sichuan Dialect recognizes discrete emotions in dialect speech but involves no articulation and no learners.

Research Program

The system has three parts.

Articulatory inversion. The first part estimates tongue-tip, blade, dorsum and lip trajectories from audio. We reuse the method of Speech Emotion Recognition with Phonation Excitation Information and Articulatory Kinematics: a temporal convolutional network on HuBERT-large features, trained on the articulography recordings in STEM-E²VA. That paper also constrains our design. Estimated kinematics added only 2.72 points to emotion recognition, against 8.45 points for measured kinematics, so inversion is noisy. We therefore treat estimated positions as uncertain and model distributions rather than point targets. We also report every result separately on measured and estimated kinematics wherever measured kinematics exist.

Continuous arousal estimation. The second part estimates arousal from the utterance as a continuous value. The idea of continuous affect dimensions comes from the valence–arousal–dominance modelling in PoliStance-VAE, where removing the continuous affect module cost 15.45% on P-STANCE. We borrow only the idea, not the text architecture. That paper also found its reconstruction decoder added little on short inputs, which is a reason not to import its full variational machinery.

Conditioned target model. The third part is a model, per phone, of where the articulators should be. For a phone $p$, an accepted variant $v$ drawn from a small set $V_p$, and estimated arousal $a$, it predicts a Gaussian over articulator positions with mean $\mu(p,v,a)$ and covariance $\Sigma(p,v,a)$. The mean shifts linearly in $a$, with the shift learned from the parallel emotional recordings. For an observed trajectory segment $x$, the error score is

$$ s(x) = \min_{v \in V_p} \; -\log \mathcal{N}\big(x;\ \mu(p,v,a),\ \Sigma(p,v,a)\big), $$

and the feedback vector is $\Delta = \mu(p, v^\star, a) - x$, where $v^\star$ is the variant that achieves the minimum. In plain terms, the learner is compared against the closest legitimate way of saying the sound at their current arousal. Only that leftover gap is turned into an instruction such as "tongue higher" or "lips more rounded."

The variant sets $V_p$ come from dialect phonology. They are seeded from expert lists of known substitutions, and the ADI-20 errors suggest the adjacent varieties that need covering. Speaker anatomy is handled with the four-corner-vowel calibration from Visualization for improving foreign language pronunciation.

Data. No existing corpus combines learners, dialects, emotion and error labels, so we build one evaluation set and are explicit that this is new collection. It covers roughly 40 adult Mandarin-L1 learners of English from Sichuan-speaking and Beijing-speaking regions, plus 10 native English speakers of two regional varieties. Each speaker reads speechocean762-style prompts under a neutral condition and under a mild arousal induction (timed, evaluated reading). Expert phoneticians label each flagged phone as error, dialect variant, or neither. This gives a three-way ground truth that no current benchmark provides.

Baselines

Naive reference. The do-nothing point is the standard Goodness of Pronunciation (GOP) score with a regression head. GOP measures how confident a speech recognizer is that the intended phone was spoken. The GOP plus support-vector-regression baseline in speechocean762 reports phone-level Pearson correlation of 0.45 and MSE of 0.16 with expert scores. We retrain it on the same speechocean762 training split our system uses.

Strongest existing detector. The strongest competitor is HMamba with decoupled cross-entropy loss, from Towards Efficient and Multifaceted Computer-assisted Pronunciation Training. It reports mispronunciation detection and diagnosis F1 of 63.85% and phone error rate of 2.72% on speechocean762. It is the best phone-level detector in the file and uses the same self-supervised features we do. We retrain it from the released recipe on identical features and data, with the same hyperparameter-search budget as our model.

Beating HMamba has two parts. First, our system must not fall meaningfully below its speechocean762 F1, which we define as no more than 2 points worse. Second, it must cut the rate at which dialect variants and emotional speech are flagged as errors on our new set by a substantial margin, defined as at least a 30% relative reduction, at matched recall on true errors.

Ablations isolating the key ingredients. We run four ablations. The first removes the variant sets, so there is a single target. The second replaces continuous arousal with the categorical emotion label. The third removes affect conditioning entirely. The fourth computes the same conditioned residual in HuBERT feature space instead of articulatory space. This last ablation tests whether articulatory space is needed for detection, or only for producing instructions.

Why not just…?

Why not just add dialect variants to the pronunciation lexicon? Supplying multiple canonical phone sequences per word, and accepting any of them, is the obvious extension of the speechocean762 voting procedure. For variants that are whole-phone substitutions, it may work as well as our model. That is a real threat. It cannot represent variants that differ below the phone level, such as vowel quality shifts within one phoneme category, and it gives no articulatory instruction. We implement it as a baseline and report results separately on substitution-type and sub-phonemic variants. If it matches us on both, the dialect half of our claim is not needed.

Why not just detect emotion and suppress feedback when the learner is aroused? Gating feedback with an off-the-shelf emotion recognizer is cheap. It also discards every real error in an aroused utterance, and anxious learners may be exactly the ones who need help. We compare gating against our correction method by error recall in the arousal condition. If our first experiment shows that false alarms do not increase with arousal, affect conditioning is unnecessary and we drop it rather than defend it.

Why not just fine-tune HMamba on dialect-diverse data? With enough labeled dialect learner data this could absorb variants implicitly. Such data does not exist in the file. ADI-20 has dialect labels but no pronunciation labels, and Whisper-large falls to 62.74% F1 on its unseen Casablanca set, a sign that implicit dialect knowledge is fragile out of domain. We fine-tune HMamba on the same small adaptation split our variant sets are built from, so both methods use equal supervision.

Why not just stay in acoustic or self-supervised feature space, given that inversion is noisy? This is the ablation described above. If feature-space residuals detect equally well, we keep articulatory space only for wording the feedback. We would then say so plainly.

Proposed Experiments

Experiment 1: false-alarm probe (the gate for everything else). We run the HMamba and GOP detectors on native speech where, by construction, there should be almost no errors: the 941 American English recordings in the Kim et al. articulography corpus. We plot flag rate against emotion category and against the 1–5 perceived-strength rating. No reported result found on Lacuna for mispronunciation false-alarm rates on emotional native speech.

The affect component survives only if flag rate rises with strength within a category. That within-category trend is the specific evidence for continuous over categorical conditioning. A parallel probe on native dialect speakers in our new set tests the dialect component.

Experiment 2: main evaluation. On the new three-way-labelled set, we measure true-error F1, dialect false-alarm rate and affect false-alarm rate for all baselines and ablations. Accuracy on speechocean762 is a required check that we do not regress.

Experiment 3: instruction quality. For STEM-E²VA and Kim et al. utterances, where measured kinematics exist, we check whether the direction of the feedback vector $\Delta$ agrees with the direction computed from measured trajectories.

Experiment 4: small learner study. On the scale of Dhvani's 22-user trial, we test whether learners given residual-based instructions improve on true errors without drifting away from their own dialect variants.

Abandonment criterion. If Experiment 1 shows flat false-alarm rates for both affect and dialect, the premise is false and the project stops there.