Abstract
Automatic Speech Recognition (ASR) systems often struggle with the high variability found in dysarthric and elderly speech. This paper introduces a novel speaker adaptation framework that utilizes "spectro-temporal deep features." By applying Singular Value Decomposition (SVD) to Mel-spectrograms, the method isolates spectral and temporal characteristics, which are then processed by a deep neural network to generate compact speaker embeddings. Experimental results on the UASpeech dataset show a new state-of-the-art Word Error Rate (WER) of 25.05%, representing a significant improvement over existing adaptation techniques.
Introduction
While ASR performance has improved dramatically for typical speech, it remains suboptimal for individuals with speech impairments or age-related changes. The primary challenge is the extreme heterogeneity among speakers; conditions like cerebral palsy or ALS manifest differently in every individual, affecting volume, clarity, and speaking rate. Traditional adaptation techniques like i-vectors often fail to capture these fine-grained pathological nuances. This research addresses the data scarcity and variability issues by proposing a multi-stage feature extraction process designed to capture the unique spectro-temporal signatures of disordered speech.
Methodology
The proposed system follows a structured pipeline to create descriptive speaker embeddings, as illustrated in Figure 1.
Spectro-Temporal Decomposition
The core of the approach lies in using Singular Value Decomposition (SVD) on the Mel-spectrogram. This mathematical operation separates the speech signal into two distinct components:
- Spectral Basis ($U$): Captures time-invariant characteristics such as the spectral envelope, which relates to vocal quality and volume.
- Temporal Basis ($V$): Captures time-variant patterns like speaking rate, rhythm, and dysfluencies.
Figure 2 visualizes this decomposition for different intelligibility levels, highlighting how SVD can isolate these critical features.
Deep Feature Extraction and Integration
The SVD bases are fed into a Deep Neural Network (DNN) classifier trained to predict speaker-specific traits such as intelligibility level or age group. Rather than using the final classification, the system extracts 25-dimensional features from a "bottleneck" layer. These embeddings provide a compressed summary of the speaker's pathological or age-related traits. Finally, these deep features are concatenated with standard acoustic features and integrated into state-of-the-art ASR architectures, including Hybrid DNN/TDNN and Conformer models.
Experimental Setup
The method was evaluated across four diverse datasets:
- UASpeech: Isolated-word dysarthric speech (English).
- TORGO: Words and short sentences from dysarthric speakers (English).
- DementiaBank Pitt: Interviews with elderly participants, including those with Alzheimer’s (English).
- JCCOCC MoCA: Cantonese elderly speech dataset.
Results and Discussion
The proposed system achieved a 25.05% WER on the UASpeech test set, setting a new state-of-the-art benchmark. Key findings include:
- Superiority over Baselines: The spectro-temporal deep features outperformed standard i-vector and x-vector adaptation by up to 2.63% absolute (8.63% relative) in WER reduction.
- Intuitive Feature Separation: SVD acts as an effective prior by separating "what" is said from "how" it is said, aligning well with the clinical symptoms of dysarthria.
- Robustness: Improvements remained even after applying other advanced adaptation techniques like LHUC (Learning Hidden Unit Contributions), suggesting that these features provide unique information.
- Architecture Agnostic: The gains were consistent across both traditional hybrid models and modern end-to-end Transformers.
Conclusion
This work demonstrates that isolating and specifically modeling the spectro-temporal characteristics of disordered speech can significantly enhance ASR performance. While the current implementation relies on speaker-level averaging (offline), future research will target real-time, "on-the-fly" adaptation to make these systems more practical for assistive technology.
Summary updated