Speaker Adaptation Using Spectro-Temporal Deep Features for Dysarthric and Elderly Speech Recognition

Mengzhe Geng, Xurong Xie, Zi Ye, Tianzi Wang, Guinan Li, Shujie Hu, Xunying Liu, Helen Meng

Abstract

Automatic Speech Recognition (ASR) systems often struggle with the high variability found in dysarthric and elderly speech. This paper introduces a novel speaker adaptation framework that utilizes "spectro-temporal deep features." By applying Singular Value Decomposition (SVD) to Mel-spectrograms, the method isolates spectral and temporal characteristics, which are then processed by a deep neural network to generate compact speaker embeddings. Experimental results on the UASpeech dataset show a new state-of-the-art Word Error Rate (WER) of 25.05%, representing a significant improvement over existing adaptation techniques.

Introduction

While ASR performance has improved dramatically for typical speech, it remains suboptimal for individuals with speech impairments or age-related changes. The primary challenge is the extreme heterogeneity among speakers; conditions like cerebral palsy or ALS manifest differently in every individual, affecting volume, clarity, and speaking rate. Traditional adaptation techniques like i-vectors often fail to capture these fine-grained pathological nuances. This research addresses the data scarcity and variability issues by proposing a multi-stage feature extraction process designed to capture the unique spectro-temporal signatures of disordered speech.

Methodology

The proposed system follows a structured pipeline to create descriptive speaker embeddings, as illustrated in Figure 1.

Overall architecture of the proposed speech recognition system with spectro-temporal subspace decomposition.

Spectro-Temporal Decomposition

The core of the approach lies in using Singular Value Decomposition (SVD) on the Mel-spectrogram. This mathematical operation separates the speech signal into two distinct components:

  • Spectral Basis ($U$): Captures time-invariant characteristics such as the spectral envelope, which relates to vocal quality and volume.
  • Temporal Basis ($V$): Captures time-variant patterns like speaking rate, rhythm, and dysfluencies.

Figure 2 visualizes this decomposition for different intelligibility levels, highlighting how SVD can isolate these critical features.

Visualization of SVD-based subspace decomposition for intelligible (INV) and partially intelligible (PAR) speech.

Deep Feature Extraction and Integration

The SVD bases are fed into a Deep Neural Network (DNN) classifier trained to predict speaker-specific traits such as intelligibility level or age group. Rather than using the final classification, the system extracts 25-dimensional features from a "bottleneck" layer. These embeddings provide a compressed summary of the speaker's pathological or age-related traits. Finally, these deep features are concatenated with standard acoustic features and integrated into state-of-the-art ASR architectures, including Hybrid DNN/TDNN and Conformer models.

Experimental Setup

The method was evaluated across four diverse datasets:

  • UASpeech: Isolated-word dysarthric speech (English).
  • TORGO: Words and short sentences from dysarthric speakers (English).
  • DementiaBank Pitt: Interviews with elderly participants, including those with Alzheimer’s (English).
  • JCCOCC MoCA: Cantonese elderly speech dataset.

Results and Discussion

The proposed system achieved a 25.05% WER on the UASpeech test set, setting a new state-of-the-art benchmark. Key findings include:

  • Superiority over Baselines: The spectro-temporal deep features outperformed standard i-vector and x-vector adaptation by up to 2.63% absolute (8.63% relative) in WER reduction.
  • Intuitive Feature Separation: SVD acts as an effective prior by separating "what" is said from "how" it is said, aligning well with the clinical symptoms of dysarthria.
  • Robustness: Improvements remained even after applying other advanced adaptation techniques like LHUC (Learning Hidden Unit Contributions), suggesting that these features provide unique information.
  • Architecture Agnostic: The gains were consistent across both traditional hybrid models and modern end-to-end Transformers.

Conclusion

This work demonstrates that isolating and specifically modeling the spectro-temporal characteristics of disordered speech can significantly enhance ASR performance. While the current implementation relies on speaker-level averaging (offline), future research will target real-time, "on-the-fly" adaptation to make these systems more practical for assistive technology.

Summary updated

Abstract

Despite the rapid progress of automatic speech recognition (ASR) technologies targeting normal speech in recent decades, accurate recognition of dysarthric and elderly speech remains highly challenging tasks to date. Sources of heterogeneity commonly found in normal speech including accent or gender, when further compounded with the variability over age and speech pathology severity level, create large diversity among speakers. To this end, speaker adaptation techniques play a key role in personalization of ASR systems for such users. Motivated by the spectro-temporal level differences between dysarthric, elderly and normal speech that systematically manifest in articulatory imprecision, decreased volume and clarity, slower speaking rates and increased dysfluencies, novel spectro-temporal subspace basis deep embedding features derived using SVD speech spectrum decomposition are proposed in this paper to facilitate auxiliary feature based speaker adaptation of state-of-the-art hybrid DNN/TDNN and end-to-end Conformer speech recognition systems. Experiments were conducted on four tasks: the English UASpeech and TORGO dysarthric speech corpora; the English DementiaBank Pitt and Cantonese JCCOCC MoCA elderly speech datasets. The proposed spectro-temporal deep feature adapted systems outperformed baseline i-Vector and x-Vector adaptation by up to 2.63% absolute (8.63% relative) reduction in word error rate (WER). Consistent performance improvements were retained after model based speaker adaptation using learning hidden unit contributions (LHUC) was further applied. The best speaker adapted system using the proposed spectral basis embedding features produced the lowest published WER of 25.05% on the UASpeech test set of 16 dysarthric speakers.
BibTeX
@article{geng2022speaker,
  author = {Mengzhe Geng and Xurong Xie and Zi Ye and Tianzi Wang and Guinan Li and Shujie Hu and Xunying Liu and Helen Meng},
  title = {{Speaker Adaptation Using Spectro-Temporal Deep Features for Dysarthric and Elderly Speech Recognition}},
  journal = {IEEE ACM Trans. Audio Speech Lang. Process.},
  year = {2022},
  url = {https://openreview.net/forum?id=ikMgCIk4o9},
}
Figure