Latent Phrase Matching for Dysarthric Speech

Colin Lea, Dianna Yee, Jaya Narain, Zifang Huang, Lauren Tooley, Jeffrey P. Bigham, Leah Findlater

Abstract

Standard Automatic Speech Recognition (ASR) systems often fail users with dysarthria due to the high degree of idiosyncratic phonetic variation in their speech. This paper presents Latent Phrase Matching (LPM), a personalized phrase recognition system that utilizes a query-by-example (QBE) framework. By leveraging low-dimensional latent embeddings and Dynamic Time Warping (DTW), LPM achieves significant performance gains over commercial ASR, particularly for individuals with severe speech impairments, while requiring minimal enrollment data.

Introduction

Automatic Speech Recognition (ASR) has become a primary interface for modern technology, yet it remains inaccessible for many individuals with speech disabilities. Dysarthria—a motor speech disorder resulting from conditions such as ALS, cerebral palsy, or stroke—causes speech patterns that deviate significantly from the "typical" datasets used to train commercial models. Traditional ASR relies on the assumption that words are composed of consistent sub-units (phones). For dysarthric speakers, these pronunciations are often highly unique, making phone-based decoding unreliable.

While large-scale personalized ASR models have shown promise, they typically require extensive datasets of atypical speech to fine-tune. This research introduces Latent Phrase Matching (LPM), a pragmatic, low-data solution designed to recognize a specific set of user-defined phrases using only a few enrollment samples.

Methodology

LPM shifts the paradigm from speech transcription to template matching. The system is composed of three primary components: an embedding model, an enrollment process, and a run-time matching engine.

1. Speech Embedding Model

The core of the system is a ResNet-style temporal convolutional network (TCN) trained on the Multilingual Spoken Words Corpus (MSWC). This model is designed to extract 128-dimensional "latent embeddings" at a rate of 100Hz. These embeddings are trained to capture the structural characteristics of speech while remaining agnostic to specific phonetic content or language-specific rules.

2. Enrollment and Template Creation

To use the system, a user records a target phrase (e.g., "Turn on the lights") at least twice. These recordings are processed by the embedding model to create reference templates ($Z_i$). This "query-by-example" approach allows the system to learn the user's specific vocal patterns without needing to map them to a standard dictionary.

3. Dynamic Time Warping (DTW) Matching

During inference, live audio is converted into embeddings and compared against the stored templates. The system employs Dynamic Time Warping (DTW) to calculate similarity. DTW is essential for dysarthric speech as it accounts for non-linear variations in speaking rate and cadence. If the similarity score falls below a predefined "within-phrase" threshold, the system triggers a detection.

Experimental Evaluation

The researchers evaluated LPM using two distinct datasets to test its robustness and generalizability:

  • Internal Dataset: Included 32 participants with varying degrees of dysarthria (mild to severe). The test set comprised 50 unique phrases recorded across different sessions and microphone distances.
  • EasyCall Dataset: A public dataset featuring Italian speech from 31 dysarthric and 25 control participants. This was used to verify the system's language-agnostic capabilities, as the embedding model was trained primarily on English.

Results and Discussion

The experimental results demonstrate that LPM significantly outperforms standard ASR in personalized contexts:

  • Performance in Severe Cases: On the internal dataset, LPM provided a 60% improvement in recall compared to commercial ASR for users with severe dysarthria. While standard ASR performance often drops to near 0% for severe impairments, LPM maintained approximately 70% accuracy.
  • Cross-Language Generalization: On the Italian EasyCall dataset, LPM improved accuracy by 30.5% over the ASR baseline, confirming that the latent representations are effective across different languages without retraining.
  • Consistency over Correctness: The primary insight of this work is that for personalized commands, the system does not need to understand "correct" phonetics; it only needs to recognize that a user produces a specific sound pattern consistently.

Limitations and Future Work

Despite its success, LPM faces challenges regarding the "Aggressor" problem—a higher False Detection Rate (FDR) compared to ASR. Because the system looks for specific patterns rather than decoding a vast vocabulary, it can occasionally misinterpret conversational background noise as a command. Additionally, performance can fluctuate based on a user's physical state (e.g., fatigue) or changes in the acoustic environment.

Future research will focus on improving robustness to intra-individual variability and refining thresholding techniques to reduce false alarms in noisy, real-world settings.

Conclusion

Latent Phrase Matching offers a high-performance, low-latency, and language-agnostic solution for dysarthric speech recognition. By focusing on consistency rather than standard phonetic accuracy, LPM provides a viable path toward making voice-controlled technology more inclusive for individuals with severe speech differences.

Summary updated

BibTeX
@inproceedings{lea2023latent,
  author = {Colin Lea and Dianna Yee and Jaya Narain and Zifang Huang and Lauren Tooley and Jeffrey P. Bigham and Leah Findlater},
  title = {{Latent Phrase Matching for Dysarthric Speech}},
  booktitle = {Interspeech},
  year = {2023},
  url = {https://openreview.net/forum?id=kFHbU7TftS},
}
Figure