An Interpretable Acoustic Coordinate System for Japanese Consonant Morae in Dysarthric Speech

Yuma Enomoto, Misa Yamaguchi, Hiroki Nishikawa, Ittetsu Taniguchi, Kazuhide Matsunaga, Narikazu Uzawa, Takao OnoyearXiv 2026 · arXiv

TL;DR

Researchers have developed a 2D visualization system that maps Japanese consonant pronunciation onto a "coordinate system" of temporal and spectral deviations. By decomposing speech errors into timing and sound quality axes, the system provides granular, interpretable feedback that helps patients with dysarthria track subtle articulatory improvements that traditional speech recognition often misses.

What Problem Does This Paper Address?

Dysarthria is a motor speech disorder—often resulting from neurological conditions or surgical resections (like oral cancer treatment)—that makes speech difficult to understand. For patients, rehabilitation is a long road. Current clinical feedback usually relies on a therapist’s subjective judgment or binary "correct/incorrect" results from Automatic Speech Recognition (ASR) systems.

Neither of these is ideal for self-directed practice. ASR doesn't tell you how you were wrong, and categorical labels (e.g., "moderate severity") don't change quickly enough to motivate a patient making small, incremental progress. While researchers have successfully visualized vowel deviations before, consonants are much harder to map because they are transient, non-stationary, and easily "smoothed over" by standard machine learning models. This paper tackles the challenge of creating a physically meaningful, 2D space specifically for Japanese consonant morae (the basic timing units of Japanese speech).

Methods

The authors propose a framework that transforms complex acoustic signals into a simple 2D plot where the origin $(0,0)$ represents "perfectly healthy" speech.

  1. Mora Extraction: Using the Montreal Forced Aligner (MFA), the system identifies the start and end of specific consonant-vowel units (morae).
  2. Temporal Distance ($d_{time}$): This measures rhythmic accuracy. It calculates the absolute natural logarithm of the ratio between the patient’s mora duration and a healthy reference average. This ensures that being "too fast" or "too slow" is treated as an equivalent rhythmic error.
  3. Spectral Distance ($d_{freq}$): To capture structural articulatory errors (like tongue placement), the team used the WavLM model—a self-supervised speech encoder. They specifically extracted features from the 6th Transformer layer, which has been shown to preserve physical vocal tract properties better than deeper layers that abstract speech into text.
  4. Alignment: They used Dynamic Time Warping (DTW) with cosine similarity to compare the patient's spectral features against healthy references, ensuring the distance calculation isn't "polluted" by minor timing differences.
  5. Visualization: The two distances become $(x, y)$ coordinates. A patient can see at a glance if their error is primarily a timing issue (vertical shift) or a spectral/postural issue (horizontal shift).

Datasets

The study utilized a Japanese speech dataset consisting of:

  • 17 healthy speakers (6 male, 11 female) providing reference "target" speech.
  • 5 male patients with dysarthria caused by subtotal resection of the tongue.
  • Task: Reading the "The North Wind and the Sun" passage.
  • Baselines: The proposed method was compared against Variational Autoencoders (VAE) and t-SNE (a common dimensionality reduction technique).

Key Insights and Ideas

  • Orthogonality is Key: Through perturbation tests (simulating errors by artificially stretching time or warping frequencies), the researchers proved the axes are independent. If you stretch the audio, only the temporal score changes. This "orthogonality" is what makes the feedback interpretable for a patient.
  • Intermediate Layers Matter: The choice of the 6th layer of WavLM was a "Goldilocks" move. Early layers contain too much raw noise; late layers are too focused on linguistic meaning. The middle layers capture the "physicality" of the speech.
  • Generative Models Over-Smooth: The study found that standard VAEs tend to smooth out the fine-grained spectral structures of consonants to prioritize vowels. By using a distance-based approach rather than a reconstruction-based one, the authors preserved the distinct "fingerprints" of different consonants.

Key Results

  • Consonant Separability: The system achieved a Silhouette Coefficient exceeding 0.5, significantly outperforming VAE baselines in keeping different consonant categories distinct in the 2D space.
  • Screening Accuracy: In tests with real dysarthric speech, the distance from the origin proved to be a powerful metric for identifying "unclear" pronunciations, achieving an ROC-AUC of 0.897.
  • High Discriminability: For most evaluated morae (like /ka/, /so/, /na/), the system achieved near-perfect ROC-AUC scores (1.00) in distinguishing healthy speech from different consonant categories.

Limitations and Future Work

The authors noted a few hurdles for real-world deployment:

  • Linguistic Context: Certain morae like /ta/ and /ga/ showed high variance because their acoustic signature changes depending on where they appear in a sentence (e.g., word-medial vs. sentence-final). Future versions will need "context-aware" reference selection.
  • Small Patient Cohort: The clinical evaluation was limited to five male patients with a specific etiology (tongue resection). Expanding this to a larger, more diverse group (including different types of dysarthria like ataxic or spastic) is a necessary next step.
  • Real-time Feedback: While the math works, the paper doesn't yet detail a real-time UI/UX implementation for bedside rehabilitation.

Summary updated

Abstract

Dysarthria reduces speech intelligibility, requiring self-directed rehabilitation based on fine-grained feedback to capture gradual articulatory changes. For Japanese speakers, such feedback is naturally provided in mora units due to its distinct moraic timing and consonant-vowel structure. While recent visualization methods have explored distance-based feedback, most are limited to vowels. Extending them to consonants is challenging, as generative embeddings often smooth transient spectral structures. We propose a visualization system for Japanese consonant morae that maps articulatory deviations onto a two-dimensional space defined by temporal and spectral distances from healthy reference speech. Experiments show consistent geometric separation of consonant categories, with Silhouette Coefficients exceeding 0.5 and origin-based ROC-AUC scores above 0.86. Furthermore, evaluations using real dysarthric speech achieved a high ROC-AUC of 0.897 in screening for unclear pronunciations. Perturbation tests confirm orthogonal temporal and spectral shifts, yielding a physically interpretable mapping for rehabilitation.
BibTeX
@misc{enomoto2026interpretable,
  author = {Yuma Enomoto and Misa Yamaguchi and Hiroki Nishikawa and Ittetsu Taniguchi and Kazuhide Matsunaga and Narikazu Uzawa and Takao Onoye},
  title = {{An Interpretable Acoustic Coordinate System for Japanese Consonant Morae in Dysarthric Speech}},
  year = {2026},
  eprint = {2611.19714},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.19714},
}
Figure