Evaluating Automatic Speech Recognition Systems in Comparison With Human Perception Results Using Distinctive Feature Measures

Xiang Kong, Jeung-Yoon Choi, Stefanie Shattuck-Hufnagel

Introduction: Beyond the Black Box of WER

Traditional evaluation of Automatic Speech Recognition (ASR) systems typically relies on Word Error Rate (WER) or Phone Error Rate (PER). While these metrics are useful for benchmarking, they function as "black boxes"—they indicate that a system failed, but not why or how. For instance, a system might mistake "bat" for "pat" (a voicing error) or "bat" for "cat" (a place of articulation error).

Furthermore, WER conflates the performance of the acoustic model with the language model, making it difficult to isolate how well a system actually "hears" sounds. This research addresses the need for a more granular, "sub-phonemic" analysis to determine if ASR systems are modeling human-like perception or merely relying on statistical shortcuts that collapse under environmental pressure, such as background noise.

Methodology

The researchers compared three distinct "listeners":

  1. Human Subjects: 20 adults tested on Vowel-Consonant-Vowel (VCV) syllables across various noise levels.
  2. HMM-based ASR: A traditional Hidden Markov Model system built using the HTK toolkit.
  3. DNN-based ASR: A modern Deep Neural Network system built using the Kaldi toolbox.

To analyze performance, they employed Linguistic Distinctive Features. Instead of simply checking phone accuracy, they evaluated:

  • Manner: Is the sound a stop, a fricative, or a nasal?
  • Place: Is the sound produced at the lips (labial) or the roof of the mouth (alveolar)?
  • Voicing: Are the vocal cords vibrating (voiced) or not (unvoiced)?

The Hierarchy of Robustness

A central finding in human speech perception is the hierarchy of feature robustness: "voicing" is the most resilient feature (easiest to hear in noise), followed by "manner," with "place" being the most fragile. The study found that the DNN-based system exhibited a similar trend, suggesting it captures acoustic features more like a human ear than older HMM systems do.

Error rates for distinctive features (voicing, manner, place) across noise levels. The DNN system shows a human-like hierarchy where place errors are most frequent.

As shown in the feature error plots, the DNN system's performance degrades as noise increases, but it maintains the relative ordering of feature difficulty. This contrasts with HMM systems, which showed much more randomized and less consistent error patterns.

Qualitative Analysis: The "/s/" Bias

Visualization through grey-scale confusion matrices revealed a striking "aha moment" at high noise levels. While humans typically stop responding when the signal is too degraded, ASR systems began to "hallucinate" specific sounds. At 0dB SNR and below, both ASR systems began to default to high-frequency noise patterns, frequently misidentifying various consonants as the fricative /s/.

Confusion matrix at 0dB SNR. Note the dark vertical band under 's', indicating that various sounds (like 'sh') are being misclassified as the fricative /s/.

This suggests that the models rely on matching noise to the most similar high-frequency phonetic pattern rather than the robust feature extraction seen in human hearing. Additionally, in human confusion matrices, errors often appear in "blocks" (e.g., confusing all voiced stops with each other), indicating that humans successfully extract some features (like voicing) even when they lose others (like place). The HMM system lacked this consistency.

Key Results and Conclusion

  • DNN Superiority: The DNN system achieved a PER of 30.87% on the TIMIT test set, significantly outperforming the HMM system, which hovered around 65-67%.
  • Human-like Errors: The DNN's error patterns were qualitatively much closer to human patterns than the HMM's, particularly in its struggle with "place" features.
  • Noise Breakdown: While human performance stayed relatively stable until 0dB SNR before dropping sharply, ASR systems showed a more linear, gradual degradation, indicating they lack the "threshold" robustness of human hearing.

Future work aims to use "distinctive-feature-distance" as a training criterion (loss function) to force ASR systems to prioritize the same acoustic cues that humans do, potentially leading to more robust recognition in real-world environments.

Summary updated

BibTeX
@inproceedings{kong2017evaluating,
  author = {Xiang Kong and Jeung-Yoon Choi and Stefanie Shattuck-Hufnagel},
  title = {{Evaluating Automatic Speech Recognition Systems in Comparison With Human Perception Results Using Distinctive Feature Measures}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2017},
  doi = {10.1109/icassp.2017.7953270},
}
Figure