Detecting Speech Abnormalities With a Perceiver-Based Sequence Classifier that Leverages a Universal Speech Model

Hagen Soltau, Izhak Shafran, Alex Ottenwess, Joseph R. Duffy, Rene L. Utianski, Leland R. Barnard, John L. Stricker, Daniela A. Wiepert, David T. Jones, Hugo Botha

Introduction

Neurological conditions such as Amyotrophic Lateral Sclerosis (ALS), Parkinson’s disease, and various forms of ataxia frequently manifest through subtle but distinct alterations in speech patterns. These abnormalities—ranging from irregular rhythms and slurred syllables to breathiness—serve as critical biomarkers for diagnosis and disease progression monitoring. Traditionally, Speech-Language Pathologists (SLPs) conduct these evaluations. However, the specialized expertise required is a scarce resource, often leading to significant diagnostic delays and infrequent monitoring.

Automating the detection of these speech abnormalities presents a significant challenge for machine learning. Clinical datasets are inherently small, often comprising only hundreds of speakers, which pales in comparison to the millions of hours used to train modern foundation models. Furthermore, the acoustic signals of interest are "sparse"; a specific symptom might only appear in the pitch of a single vowel or the timing between specific consonants. Standard sequence models that aggregate or average an entire audio clip often fail to capture these localized, critical details.

Methodology

This research introduces a high-performance system developed by Google DeepMind and the Mayo Clinic. The architecture is designed to be both data-efficient and highly sensitive to long-range, localized temporal patterns through a two-stage approach.

The Encoder: Universal Speech Model (USM)

To overcome data scarcity, the system leverages a "frozen" Universal Speech Model (USM) as a feature extractor. The USM is a 2-billion parameter model pre-trained on 12 million hours of audio across 300 languages. By using a pre-trained foundation model, the system benefits from a rich, high-dimensional representation of speech acoustics that would be impossible to learn from clinical data alone.

The Classifier: Perceiver-based Sequence Modeling

The core innovation lies in the downstream classifier. Unlike traditional Transformers that utilize a single "summary token" (like a [CLS] token) to represent an entire sequence, this model employs a Perceiver-based architecture with class-specific latent variables.

For each of the 14 medical attributes being tracked (e.g., "tremor," "syllable segmentation," or "harshness"), the model maintains a dedicated set of learned "queries." Through a cross-attention mechanism, these queries selectively attend to the USM encodings. This allows the model to focus on different regions of the audio signal simultaneously, extracting only the information relevant to a specific symptom.

Factorized Projection

To finalize predictions, the model utilizes a factorized linear layer. This layer operates across both the temporal and feature dimensions, enabling the system to capture joint dependencies—such as how a specific pitch change over a specific duration indicates a particular abnormality—without the computational overhead of a full dense layer.

Experimental Setup and Datasets

The model was evaluated using a curated corpus from the Mayo Clinic, consisting of 2,594 audio files from 782 unique speakers. The evaluation focused on three standard clinical speech tasks:

  1. Vowel Prolongation (VP): Sustaining an "ah" sound to check for stability and breathiness.
  2. Alternate Motion Rate (AMR): Rapidly repeating a single syllable (e.g., "puh-puh-puh") to assess rhythm and speed.
  3. Sequential Motion Rate (SMR): Rapidly repeating a sequence of syllables (e.g., "puh-tuh-kuh") to evaluate articulatory coordination.

Results and Discussion

The proposed system achieved a state-of-the-art average accuracy of 83.1%, significantly outperforming standard Transformer architectures (80.9%) and conventional Perceiver models (81.8%). Several key insights emerged from the ablation studies:

  • The Value of ASR Pre-training: Interestingly, models pre-trained for Automatic Speech Recognition (ASR)—learning what was said—outperformed models trained only on reconstruction tasks. This suggests that the phonetic boundaries learned during ASR training are highly relevant to identifying pathological deviations in speech.
  • Optimal Layer Selection: The researchers identified a "Goldilocks" zone within the USM. The middle layers (specifically around layer 11 of 32) yielded the highest diagnostic accuracy. Early layers were found to be too focused on raw acoustics, while the deepest layers were overly specialized for linguistic content. The middle layers provided the ideal balance of acoustic and phonetic information.
  • Foundation Model Impact: Transitioning from standard log-mel spectrograms to USM encodings resulted in a performance jump from 76.4% to 83.1%, highlighting the necessity of large-scale pre-training for small-data clinical tasks.

Conclusion

By combining the broad representational power of a Universal Speech Model with a specialized, attention-driven Perceiver classifier, this research demonstrates a viable path toward automated clinical speech assessment. While the results are promising, future work will focus on multi-layer fusion techniques to capture symptoms that manifest at different levels of the acoustic hierarchy and exploring the integration of these models into real-world clinical workflows to support healthcare providers.

Summary updated

BibTeX
@inproceedings{soltau2023detecting,
  author = {Hagen Soltau and Izhak Shafran and Alex Ottenwess and Joseph R. Duffy and Rene L. Utianski and Leland R. Barnard and John L. Stricker and Daniela A. Wiepert and David T. Jones and Hugo Botha},
  title = {{Detecting Speech Abnormalities With a Perceiver-Based Sequence Classifier that Leverages a Universal Speech Model}},
  booktitle = {ASRU},
  year = {2023},
  doi = {10.1109/asru57964.2023.10389789},
}
Figure