Systems that flag mispronounced sounds in a learner's or patient's speech fail in a specific way. They know what the speaker was supposed to say, and that knowledge pulls them toward hearing the expected sound even when a different one was produced. We propose to break this pull with a loop. First, the detector makes a soft guess about where in time the speech departs from the expected pronunciation. It then re-reads the audio with its trust in the expected text, and in its most "expectation-laden" internal features, turned down inside those suspected regions. Between passes, the guess is cleaned up by a small convex optimization over time: a smoothing problem with one global optimum, whose rounded solution is provably the best binary segmentation for its objective. Repeating this two or three times should catch errors that a single pass explains away.
The opportunity comes from putting together two findings that have not yet been connected. The text-aware work of Text-Aware End-to-end Mispronunciation Detection and Diagnosis showed that naive attention from reference text to audio "forcibly aligns" during mispronunciations and hides the error. Vision work on feeding a model's own mask back as input, Iterative Instance Segmentation, showed that a network given its previous hypothesis can repair mistakes it made in a single shot. Our proposal uses the fed-back mask to decide where the text prior should be withheld. That is a job a one-pass gate cannot do, because the gate would need to know where the errors are before it has found them.
Core Claim
Mispronunciation detection can be framed as predicting a relaxed error mask $m \in [0,1]^T$ over the $T$ audio frames. Here $m_t$ is the degree to which frame $t$ departs from the canonical phone. Two conditions matter. First, the mask must be refined iteratively. Second, the mask must be used to suppress the reference-text conditioning and the upper self-supervised layers exactly where it is high. When both hold, detection recall rises on substitutions and subtle distortions without a matching loss of precision. These gains should exceed what the same compute buys as a deeper single pass, and what temporal smoothing buys as a post-process. The continuous relaxation is what makes the loop stable and interpretable. The multi-scale ingredient, mixing lower acoustic layers with upper linguistic layers, is the thing the mask acts on.
Research Program
The base detector is a wav2vec 2.0 or WavLM encoder that receives the canonical phone sequence, in the style of the text-aware gated model above. At step $k$ it reads three inputs: the audio, the reference phones, and the previous mask $m^{(k)}$, which is initialized to a neutral 0.5 following Iterative Instance Segmentation. The mask has two effects. It multiplies the text-conditioning pathway by $(1-m_t)$, so frames suspected of error are decoded mostly from acoustics. It also shifts a per-frame mixture over encoder layers toward lower layers. The layer choice follows the probing results of Phoneme Segmentation Using Self-Supervised Speech Models, which show that frozen layer-specific readouts carry precise phone-boundary information. Our hypothesis is that upper layers carry the phonotactic expectations behind the auto-correction trap. The network emits per-frame error probabilities $p^{(k)}_t$ and, as a diagnosis head, the phone actually produced.
Between steps, the raw probabilities are not fed back directly. They pass through a one-dimensional total-variation (TV) problem, which penalizes the number and size of jumps in the mask:
$$m^{(k+1)} = \arg\min_{m \in [0,1]^T} \tfrac12 \sum_{t} \big(m_t - p^{(k)}_t\big)^2 + \lambda \sum_{t} |m_{t+1} - m_t|,$$
Here $\lambda$ sets how strongly short spurious error blips are merged or removed. This is the continuous relaxation idea of the variational segmentation line, such as the fuzzy-membership TV model of Variational Segmentation using Fuzzy Region Competition and Local Non-Parametric Probability Density Functions and the lifted convex formulations of Cremers et al., transplanted to a 1D acoustic signal. The file identifies that setting as largely unexplored. The 1D case avoids the memory and lifting costs flagged for images and volumes. The problem is solved exactly in linear time, is differentiable almost everywhere for end-to-end training, and has a unique solution. By the thresholding results summarized in the file, thresholding at 0.5 gives the globally optimal two-phase error segmentation for the corresponding binary energy. This is useful when feedback is shown to learners, because the regions we highlight are the optimum of a stated objective rather than an arbitrary cut of a heuristic curve.
Training unrolls $K=3$ steps with a loss at every step. To keep the model from depending on its own clean predictions, it is trained on a mixture of its detached previous masks and randomly corrupted masks. Targets can optionally be softened using the articulatory similarity matrices of Towards Accurate Phonetic Error Detection Through Phoneme Similarity Modeling, so that near-miss substitutions are penalized less. At inference we adopt the acceptance gate of SAM-REF: a refined mask is kept only if the TV energy under the new evidence decreases. This guards against the drift that the file lists as the main failure of ungated feedback loops.
Baselines
The naive reference point is the fine-tuned wav2vec 2.0 binary detector of Explore wav2vec 2.0 for Mispronunciation Detection. It reports F1 0.610 (precision 0.580, recall 0.643) on L2-ARCTIC, and alongside it the CTC-based fine-tuning of A Study on Fine-Tuning wav2vec2.0 Model for the Task of Mispronunciation Detection and Diagnosis, which reports F1 60.44% on L2-ARCTIC with TIMIT anchoring. These establish what the encoder achieves with no text prior and no loop.
The strongest existing competitor is the text-aware model with the Textual Modulation Gate, Text-Aware End-to-end Mispronunciation Detection and Diagnosis. It reports L2-ARCTIC F1 61.75%, up from 57.51% for naive text fusion. It is the right opponent because it attacks the same forced-alignment failure with a single-pass gate. We will reimplement it on our encoder, data split, and training budget, and report both our reimplementation and the published figure. Beating it means a paired improvement in F1 across at least three seeds, with the gain concentrated in recall on substitutions and distortions. A precision-driven gain would indicate that the model has merely become conservative.
For robustness beyond read L2 speech, the comparison is the soft-label similarity model of Towards Accurate Phonetic Error Detection Through Phoneme Similarity Modeling. Trained on VCTK-accent, it reports PER 26.67% on L2-ARCTIC, 30.49% on MultiPA, and 38.63% on PPA Speech. We will train on the same VCTK-accent data and evaluate on the same three test sets. For diagnosis quality, we compare against the phonological-feature model of Phonological-Level Mispronunciation Detection and Diagnosis, which reports FAR below 30% and DER below 10% on L2-ARCTIC. FAR, the fraction of real errors accepted as correct, is the direct measure of the auto-correction trap, so it is our primary secondary metric.
Four ablations isolate the key ingredient. The first is a compute-matched single-pass model with extra layers, equal in FLOPs to three passes. The second feeds the mask back as an input channel but does not let it gate the text or layer mixture. The third gates the text but not the layers, and the fourth gates the layers but not the text. The fifth removes the TV step, feeding raw probabilities back. The sixth is an oracle that feeds the ground-truth error mask, which bounds what mask-conditioned re-reading can ever deliver.
Why not just…?
Why not just smooth the one-pass error curve over time? This is the most serious threat. The same TV problem applied once to a single-pass detector's output, or an HMM-style smoother, is cheap and may capture much of any gain if the benefit of iteration is only temporal coherence. We include exactly this baseline: the one-pass text-gated detector followed by the identical TV step. The loop is justified only if re-reading beats smoothing on FAR and substitution recall. That outcome requires the second pass to recover acoustic evidence that the first pass suppressed. Smoothing can only redistribute evidence the first pass already produced, so it cannot recover suppressed evidence.
Why not just train with soft phoneme-similarity labels? The similarity-matrix approach cut PER on VCTK-accent from 18.31% to 12.37%. That shows softer targets help. But it changes what the model is rewarded for, not what it listens to. The auto-correction trap is a conditioning problem, and soft labels leave the text prior fully active at error sites. The two are complementary: we test our loop with and without soft targets. If soft labels alone match the full system on FAR, the conditioning story is wrong.
Why not just use a bigger or deeper single-pass model? Three passes cost roughly three times the inference compute, and the file notes that latency scales linearly with iterations. The compute-matched deeper model tests this directly. We also report accuracy against passes, since most of the gain in Iterative Instance Segmentation came within three steps. If the deeper model ties, the proposal reduces to a capacity effect and we would say so.
Why not just drop the reference text altogether? A free phone recognizer cannot be pulled toward the expected sound. However, the jump from 57.51% to 61.75% F1 for gated text fusion indicates that the prior is valuable where speech is correct. The text-free baselines above measure the cost of discarding it everywhere, rather than only where the mask says to.
Proposed Experiments
Training uses L2-ARCTIC with TIMIT native anchoring, following the wav2vec 2.0 MDD studies, and separately VCTK-accent for the generalization protocol. Evaluation covers L2-ARCTIC, MultiPA, and PPA Speech. Metrics are MDD precision, recall, and F1; FAR and DER; and PER for the diagnosis head. For localization, where phone-level time annotations are available, we add an error-region boundary F1 at a 20 ms tolerance. We report both lenient and strict hit counting, given the protocol inconsistency the file documents in segmentation work.
Two checks run first and decide whether to continue. The first is the oracle-mask experiment. If gating text and upper layers with the true error mask does not cut FAR substantially below the one-pass text-gated model, then mask-conditioned re-reading has little to offer and the project stops. The second is the smoothing-versus-rereading comparison at equal compute, which separates the value of the loop from the value of temporal regularization.
Beyond these, we will measure how often the energy gate rejects updates and whether accepted updates are monotone across steps. We will also break down gains by error type, expecting the largest gains on substitutions between articulatorily close phones. Finally, we will test whether the advantage widens on dysarthric and aphasic speech in PPA Speech, where expectations and acoustics diverge most and where the forced-alignment failure should be strongest.
References
- Text-Aware End-to-end Mispronunciation Detection and Diagnosis
- Iterative Instance Segmentation
- Phoneme Segmentation Using Self-Supervised Speech Models
- Variational Segmentation using Fuzzy Region Competition and Local Non-Parametric Probability Density Functions
- An algorithm for minimizing the Mumford-Shah functional
- Towards Accurate Phonetic Error Detection Through Phoneme Similarity Modeling
- SAM-REF: Introducing Image-Prompt Synergy during Interaction for Detail Enhancement in the Segment Anything Model
- Explore wav2vec 2.0 for Mispronunciation Detection
- A Study on Fine-Tuning wav2vec2.0 Model for the Task of Mispronunciation Detection and Diagnosis
- Phonological-Level Mispronunciation Detection and Diagnosis