Reading Distortion as the Residue of a Reduction Curve in Spontaneous Japanese

Automatic pronunciation checkers work by transcribing the sounds a speaker actually produced, lining that transcription up against the dictionary pronunciation, and flagging every mismatch as an error. On read speech from language learners this works reasonably well. On relaxed, unscripted speech it breaks down. Fluent native speakers routinely weaken consonants, drop or devoice vowels, and merge sounds when they talk fast. A mismatch against the dictionary is therefore usually not an error at all. We propose to separate the two sources of mismatch within each speaker. The first is reduction, which grows with how fast and casually the person is speaking. The second is distortion, which persists no matter how slowly they speak. For each speaker and each consonant, we fit a curve of mismatch rate against "reduction pressure" (local speaking rate plus how strongly the surrounding sounds invite weakening). We then read off where the curve lands when that pressure is extrapolated to zero. The level that remains is our distortion estimate.

The design borrows its logic from the cycloheximide chase assay in cell biology. There, the amount of a protein in a cell reflects synthesis minus degradation, and a single snapshot cannot tell the two apart. Researchers therefore block synthesis and track the protein over time to isolate the degradation term (Measuring Protein Half-Life via Translation Inhibition). The assay also teaches a caution: blocking synthesis globally stresses the cell and distorts the very rate being measured. Our version of that pitfall is asking a speaker to read aloud, which makes them over-articulate. We therefore take the "zero-pressure" reading by extrapolating across the natural variation in speaking rate inside spontaneous speech, rather than by forcing a different speaking style. The test bed is the Corpus of Spontaneous Japanese (CSJ), roughly 230 hours of unscripted lectures with phone-level alignments. Japanese devoicing and consonant weakening make CSJ an especially hard case for telling reduction from error.

Core Claim

A mismatch count taken at a single moment confounds reduction with distortion. A mismatch curve taken across reduction pressure separates them. For speaker $s$ and canonical consonant $c$, let $r$ be the reduction pressure at a token and $d \in \{0,1\}$ indicate a mismatch between the recognized and canonical phone. We model

$$ p(d=1 \mid r) = \rho_{s,c} + (1-\rho_{s,c})\left(1 - e^{-k_{s,c}\, r}\right), $$

In this equation, $\rho_{s,c}$ is the pressure-independent distortion rate and $k_{s,c}$ is how quickly reduction builds up as pressure rises. The exponential form is the chase assay's first-order kinetics carried over: each unit of added pressure removes a constant fraction of the tokens that are still realized canonically.

The claim has two parts. First, $\rho$ should be near zero for healthy native speakers even where raw mismatch rates are high. Second, genuine distortions should raise $\rho$ while leaving $k$ roughly unchanged. Spirantization, where stops fail to close fully and come out as fricatives, is one example. Study of the Performance of Automatic Speech Recognition Systems in Speakers with Parkinson's Disease documented this pattern in Parkinsonian Spanish.

To our knowledge nothing in the literature does this. The closest CSJ work, String and lattice based discriminative training for the corpus of spontaneous Japanese lecture transcription task, trains the acoustic model to separate correct transcriptions from competing ones and reports only word error rate. It never analyzes phone-level substitutions, deletions, or insertions. Mispronunciation work such as An Effective End-to-End Modeling Approach for Mispronunciation Detection and Automatic Deep Neural Network-Based Segmental Pronunciation Error Detection of L2 English Speech (L1 Bengali) evaluates on read learner speech. In that setting every mismatch is assumed to be an error.

Research Program

The pipeline has three stages.

Stage 1: soft phone recognition. We train a phone recognizer on CSJ. It is kept deliberately free of word-level language modeling, because Towards Temporally Explainable Dysarthric Speech Clarity Assessment shows that Whisper's strong text decoder turns phone deletions and insertions into word substitutions. Following the lattice-training result on CSJ, we keep the recognizer's full set of competing phone hypotheses (a lattice) rather than only its single best guess. Each canonical token then gets a soft mismatch probability, which the 1-best Needleman-Wunsch alignment used in prior mispronunciation work throws away.

Stage 2: reduction pressure. We compute $r$ per token from three quantities:

  • local articulation rate, relative to the speaker's own median;
  • a phonological licensing score, for example a high vowel between voiceless consonants for devoicing, or an intervocalic stop for lenition;
  • position within the phrase.

All three come from the CSJ alignments and the corpus's morphological layers (Morphological Annotation of a Large Spontaneous Speech Corpus in Japanese).

Stage 3: fitting the curves. We fit the model above hierarchically. Parameters for each speaker and consonant are shrunk toward values for the consonant's manner class (stop, fricative, nasal, affricate), so that sparse cells still get stable estimates. Uncertainty in $\rho$ is reported from the posterior.

The chase analogy also dictates the diagnostics. A chase assay needs enough time points to fit a decay, and our curve needs enough spread in $r$. We will therefore report, for each speaker, the range of pressure observed and how far the intercept had to be extrapolated. Speakers whose data never approach low pressure are flagged rather than scored.

Baselines

Do-nothing reference. A phone recognizer's 1-best output is aligned to the canonical sequence with Needleman-Wunsch, and every mismatch is called a distortion. This is the standard pipeline from the Bengali L2 and end-to-end CTC-Attention papers. For reference, on read learner speech An Effective End-to-End Modeling Approach for Mispronunciation Detection reports an F1 of 0.692 on the Mandarin Annotated Spoken corpus, and 0.701–0.702 with text-prompt conditioning. Automatic Deep Neural Network-Based Segmental Pronunciation Error Detection of L2 English Speech (L1 Bengali) reports an F1 of 53.19%. We will reimplement the text-prompted CTC-Attention variant on CSJ with the same acoustic training data and the same tuning budget as our recognizer, and treat it as the strongest existing method. No reported result was found on Lacuna for distortion classification on CSJ.

Goodness of Pronunciation (GOP). GOP scores each phone by the acoustic model's posterior for the intended sound. We include it because it remains the default in deployed tools, even though L2-ARCTIC: A Non-native English Speech Corpus reports a precision and recall of only 0.29 for substitutions.

Whisper with phone alignment. We run Whisper and phonemize its output as the foundation-model competitor.

Context-only classifier. This baseline receives exactly the same pressure features we compute, plus the lattice mismatch probability. It is trained to predict "licensed reduction vs. not" pooled across speakers, with no per-speaker curve and no extrapolation. It isolates our key ingredient.

Ablations. Two ablations remove one element each: fitting a single population curve per consonant instead of per speaker, and using 1-best instead of lattice mismatch.

What counts as beating the baselines. All systems share the same recognizer training data, and every learned baseline gets the same hyperparameter search budget. We count a win if both of the following hold: a lower false-distortion rate on healthy native CSJ speakers at matched recall on injected distortions, and higher macro-F1 on the adjudicated subset described below. Both must exceed confidence intervals from a speaker-level bootstrap.

Why not just…?

Why not just use GOP with a stricter threshold? A threshold on the posterior cannot tell a devoiced vowel that the speaker's rate licenses from one that is anomalous. It sees the same weak posterior in both cases. GOP also has no natural handling of deletions and insertions, as the L2-ARCTIC results show. The injected-distortion experiment reports GOP's full precision–recall curve, so a threshold sweep is covered.

Why not just use Whisper, which already handles spontaneous speech? Whisper's word-level success does not carry over to phones. Towards Temporally Explainable Dysarthric Speech Clarity Assessment finds that phone deletions and insertions go largely undetected or are relabeled as word substitutions. Word error rates also rise from 3% to nearly 20% on unmonitored conversation (Spontaneous Speech Corpora for Conversational Audio Modeling). We still run it, because it may do well on substitution-type distortions.

Why not just train a context classifier to predict licensed reduction? This is the real threat. If phonological environment and rate explain reduction well at the population level, per-speaker extrapolation may add nothing. The context-only baseline and the pooled-curve ablation test this directly. The specific claim at stake is that speakers differ in both their baseline reduction and their distortion rate, and a pooled classifier absorbs a habitually casual speaker's reductions as distortions or vice versa. If the pooled classifier matches our method on the adjudicated subset and on the injection test, we will report that the curve framing is unnecessary.

Why not just compare each speaker's spontaneous speech to careful or read speech? This is the cycloheximide pitfall. Changing the speaking style changes articulation itself, so the reference condition is not the spontaneous articulator with reduction switched off. We test this by comparing our extrapolated $\rho$ against mismatch rates in each speaker's slowest decile of tokens. The prediction is that the slow-decile rate sits below $\rho$ for fricatives and stops, a sign of over-articulation. If it does not, the simpler slow-token reference suffices and we will adopt it.

Proposed Experiments

Adjudicated subset. Phoneticians label about two hours of CSJ lectures, stratified by speaker rate. Each recognized–canonical mismatch is marked as licensed reduction, speech error, or unclear. Metrics are macro-F1 over the three labels and the false-distortion rate on reductions.

Controlled injection. At aligned phone boundaries, we splice a speaker's own fricative tokens in place of a subset of their stops, with crossfades. This simulates spirantization at known locations and rates, and every system sees identical audio. The decisive check is whether $\rho$ for stops rises in proportion to the injection rate while $k$ stays flat. A pooled classifier cannot express that dissociation.

Clinical transfer. Using the Spanish setup of Study of the Performance of Automatic Speech Recognition Systems in Speakers with Parkinson's Disease, which trains on Fisher Spanish and tests on Neurovoz (43 PD, 46 healthy controls), we fit curves on controls and apply them to patients. The prediction is that spirantization appears as elevated $\rho$ for /p, b/. We also test whether $\rho$ correlates with time since diagnosis more strongly than the reported word-error-rate correlation of ρ = 0.47. Neurovoz consists of fixed read sentences, so the spread of pressure will be narrow. That makes this check a stress test of the extrapolation diagnostics rather than a showcase.

Recognizer sanity check. The recognizer's word accuracy should sit near the lattice-trained CSJ systems, which report 18.7% WER. This confirms that differences in distortion scoring do not come from a weak front end.