The Psychometrics of Automatic Speech Recognition

Lotte Weerts, Stuart Rosen, Claudia Clopath, Dan F. M. GoodmanbioRxiv 2021 · DOI

TL;DR

This paper investigates how well modern Automatic Speech Recognition (ASR) systems mimic human speech perception by testing them on a battery of psychoacoustic experiments. While some ASR systems show promising similarities to humans in certain aspects, none fully replicate human auditory processing, highlighting key differences in how they utilize spectral, temporal, and periodicity cues.

What Problem Does This Paper Address?

The field of neuroscience has seen success in using deep neural networks (DNNs) as models for visual processing. This has led to interest in extending this approach to the auditory system, specifically for speech recognition. The core problem this paper addresses is whether current state-of-the-art ASR systems, particularly those based on DNNs, can serve as reliable proxies for human speech recognition (HSR). This is crucial because if ASR systems behave similarly to humans, they could be valuable tools for generating and testing hypotheses about human hearing. However, it's not guaranteed that ASR systems, trained on different objectives and data, will process speech in the same way humans do, especially when faced with distorted or noisy audio.

Methods

The researchers employed a comprehensive test battery of psychometric experiments, mirroring those used in human auditory research. They systematically compared the performance of three publicly available ASR systems (Kaldi nnet3 chain, Mozilla DeepSpeech, and Facebook Wav2Vec 2.0) against human performance on various auditory dimensions. These tests included:

  • Spectral Invariance: Assessing sensitivity to bandpass filtering to understand how ASR systems utilize spectral redundancy.
  • Peak and Centre Clipping: Evaluating robustness to nonlinear distortions common in communication systems.
  • Spectral and Temporal Modulations: Investigating the reliance on fluctuations in sound's frequency and amplitude.
  • Temporal Fine Structure (TFS): Examining the use of rapid fluctuations in the narrowband carrier of speech.
  • Target Periodicity: Assessing how ASR systems process the periodic nature of voiced speech.
  • Competing Talker Backgrounds: Evaluating performance in noisy environments with multiple speakers.
  • Masker Modulations and Periodicity: Analyzing the impact of masker characteristics on speech intelligibility.

The ASR systems were tested in a "mismatched" condition, meaning they were trained on clean speech but tested on distorted versions, simulating real-world scenarios where ASRs encounter novel acoustic conditions. The researchers also developed an open-source Python toolbox, "HumanlikeHearing," to facilitate such comparisons.

Datasets

  • Training Data:
    • Kaldi nnet3 chain: LibriSpeech (approx. 1000h for pre-training, 100h for fine-tuning).
  • Mozilla DeepSpeech: Full LibriSpeech (3000h) plus other publicly available English data.
  • Facebook Wav2Vec 2.0: LibriSpeech (approx. 1000h for pre-training, 100h for fine-tuning).
  • Evaluation Data:
    • IEEE sentences: A freely available dataset of 100 sentences spoken by 12 native British English speakers, used for most experiments. This dataset is phonetically balanced but lacks strong semantic context.
  • CRM corpus: Used for a specific Temporal Fine Structure experiment to mimic a closed-set task.
  • Human data: Existing datasets from previous psychometric studies (e.g., Warren et al., 2000; Kates and Arehart, 2005; Elliott and Theunissen, 2009; Hopkins and Moore, 2010; Steinmetzger and Rosen, 2015; Rosen et al., 2013).

Key Insights and Ideas

  • ASRs are not yet human proxies: Despite impressive performance on clean speech, none of the tested ASR systems fully replicate human speech perception across various distortions and noise conditions.
  • Divergent cue usage: ASR systems utilize spectral invariance, temporal fine structure, and speech periodicity differently than humans.
  • Wav2Vec 2.0 leads the pack: The Wav2Vec 2.0 CNN-Transformer model, being the most recent and largest, showed the most robustness to various distortions, aligning better with human performance in several tests.
  • TFS and periodicity are human strengths: Humans rely significantly on temporal fine structure and exhibit distinct preferences for periodicity in speech, which ASR systems largely fail to exploit or process similarly.
  • Retraining has side effects: Fine-tuning ASR models on specific distortions can improve performance on that distortion but may negatively impact performance on others, highlighting the complexity of achieving general robustness.
  • Unsupervised pre-training is promising: The success of Wav2Vec 2.0 suggests that unsupervised pre-training to extract rich acoustic features is a key step towards more humanlike ASR systems.

Key Results

  • Spectral Invariance: ASR systems required a much wider spectral range than humans to achieve comparable performance, with accuracy dropping significantly for bandpass filtered speech.
  • Clipping: While ASR systems (CNN-Transformer and DNN-HMM) showed some robustness to center clipping, they performed significantly worse than humans under peak clipping.
  • Modulations: ASR systems showed some similarity to humans in their sensitivity to spectral and temporal modulations, but with a slightly higher importance of slow modulations.
  • TFS: ASR systems showed only a tiny sensitivity to TFS, with benefits disappearing when TFS was disrupted across the whole frequency range, unlike humans who show consistent benefits.
  • Periodicity: Humans perform poorly with fully periodic speech, while DNN-HMM and LSTM performed equally well across periodic and aperiodic conditions, suggesting they don't use periodicity as a cue. Wav2Vec 2.0 showed some human-like behavior here.
  • Competing Talkers: DNN-HMM and CNN-Transformer showed some human-like interactions with informational and energetic masking, but all ASRs struggled with single-speaker scenarios compared to humans.
  • Masker Periodicity: Only the CNN-Transformer showed a benefit from masker modulations and periodicity, similar to humans.

Limitations and Future Work

  • Limitations:
    • ASR systems were not trained on distorted data, leading to expected performance drops.
  • The "mismatched" condition for ASRs is not a perfect analogue to human experience, as humans have a lifetime of exposure to various acoustic conditions.
  • The lack of semantic context in the IEEE sentences might affect ASR performance differently than human performance.
  • Some ASR models (LSTM, DNN-HMM) are limited by their input features (MFCCs), which discard temporal fine structure information.
  • Future Work:
    • Continued cross-fertilization of ideas between human and ASR research to develop more humanlike ASR systems.
  • Designing ASR systems that exploit similar speech features as humans (e.g., TFS, periodicity).
  • Investigating the role of unsupervised pre-training for creating robust acoustic representations.
  • Utilizing the open-source "HumanlikeHearing" toolbox to identify new research directions and find ASR systems that can serve as better proxies for human listeners.
  • Exploring how ASR models that can focus on specific speakers might perform in competing talker scenarios.

Summary updated

Abstract

Deep neural networks have had considerable success in neuroscience as models of the visual system, and recent work has suggested this may also extend to the auditory system. We tested the behaviour of a range of state of the art deep learning-based automatic speech recognition systems on a wide collection of manipulated sounds used in standard human psychometric experiments. While some systems showed qualitative agreement with humans in certain tests, in others all tested systems diverged markedly from humans. In particular, all systems used spectral invariance, temporal fine structure and speech periodicity differently from humans. We conclude that despite some promising results, none of the tested automatic speech recognition systems can yet act as a strong proxy for human speech recognition. However, we note that the more recent systems with better performance also tend to better match human results, suggesting that continued cross-fertilisation of ideas between human and automatic speech recognition may be fruitful. Our open source toolbox allows researchers to assess future automatic speech recognition systems or add additional psychoacoustic measures.
BibTeX
@misc{weerts2021psychometrics,
  author = {Lotte Weerts and Stuart Rosen and Claudia Clopath and Dan F. M. Goodman},
  title = {{The Psychometrics of Automatic Speech Recognition}},
  year = {2021},
  doi = {10.1101/2021.04.19.440438},
  howpublished = {bioRxiv},
}
Figure