An assessment of automatic speech recognition as speech intelligibility estimation in the context of additive noise

Wei M. Liu, John S. D. Mason, Nicholas W. D. Evans, Keith A. JellymanInterspeech 2006

TL;DR

This paper investigates whether automatic speech recognition (ASR) systems can effectively predict how intelligible speech is when degraded by various real-world noises. The authors found that ASR, particularly its word accuracy and substitution/insertion metrics, significantly outperformed traditional objective quality measures like PESQ in correlating with human intelligibility judgments. This suggests ASR could serve as a powerful, more direct tool for estimating speech intelligibility in noisy environments.

What Problem Does This Paper Address?

Accurately assessing speech intelligibility is crucial for communication systems, especially in noisy environments. While objective quality measures exist, they often focus on general speech quality rather than the specific challenge of intelligibility. Previous work has shown promise for ASR in estimating intelligibility under codec distortions, but its effectiveness in the context of additive, real-world background noises remained largely unexplored. This paper aims to bridge that gap by comparing ASR's performance against established objective measures in predicting human intelligibility across a range of noisy conditions.

Methods

The researchers evaluated seven objective measures: SegSNR, CD, WSS, MNB, MBSD, PESQ, and ASR. Six of these (SegSNR to PESQ) are "intrusive," meaning they require a clean reference signal for comparison. PESQ is the current ITU standard for speech quality assessment. ASR, on the other hand, is a non-intrusive approach that directly assesses the recognizability of speech. For ASR, they specifically looked at metrics like word accuracy (ASRacc), percentage correct (ASR%correct), and error types (insertion, deletion, substitution). Human intelligibility was measured through subjective listening tests, serving as the ground truth. The performance of each objective measure was then quantified by its correlation with these human judgments.

Datasets

The study utilized the TIDigits database for both subjective and objective tests. This database contains 11 words (digits 0-9, "oh", "zero"). For degradation, they used 8 real-life background noises from the Aurora 2 digit string corpus: airport, babble, car, exhibition hall, restaurant, street, subway, and train station. These noises were added to 566 clean four-digit strings from TIDigits at signal-to-noise ratios (SNRs) ranging from -10dB to 0dB in 0.5dB intervals, resulting in 168 degradation conditions. A separate set of 8440 clean utterances was used to train the ASR recognizer.

Key Insights and Ideas

  • ASR as a direct intelligibility predictor: The most striking finding is that ASR metrics, particularly word accuracy (ASRacc) and error types like substitution (ASRsubst) and insertion (ASRins), showed the strongest correlation with human intelligibility. This suggests ASR can directly mimic human perception of intelligibility, unlike many traditional measures.
  • Traditional measures fall short: Many established objective measures, including the state-of-the-art PESQ, performed poorly, often showing negative correlations with human intelligibility. This highlights a significant gap in current objective assessment tools for intelligibility in noisy scenarios.
  • WSS shows promise, but ASR excels: While Weighted Spectral Slope (WSS) was one of the better-performing traditional measures, its correlation was still considerably lower than the top ASR metrics.
  • Not all ASR metrics are equal: The study revealed that some ASR statistics (like deletion rate, ASRdel) were not useful predictors of intelligibility, and some even showed negative correlations. This indicates a need for careful selection of ASR features for intelligibility estimation.
  • Noise type matters: The performance of objective measures varied significantly depending on the type of background noise, with some measures misinterpreting speech-like noises (e.g., babble) as less damaging than stationary noises.

Key Results

The paper reports Pearson correlation coefficients between objective measures and subjective intelligibility scores. The top-performing ASR statistics achieved correlations as high as 0.90 with human judgments (specifically, ASRsubst with SUBJdunno). WSS showed a respectable correlation of around 0.56. In contrast, PESQ had a correlation of -0.62 with SUBJdunno, indicating poor performance. The study found that ASRacc, ASRsubst, and ASRins were the most reliable ASR indicators, with correlations ranging from 0.80 to 0.88.

Limitations and Future Work

The authors acknowledge a key limitation: the subjective tests used fixed-length test signals. This prevented the identification of insertion errors and word accuracy from the human responses, as listeners knew how many digits to expect. Future work should involve repeating these tests with variable-length signals to further investigate the correlation between ASR and human transcriptions. Additionally, the study focused solely on additive background noise; further research is needed to explore ASR's applicability in other degradation contexts (e.g., channel distortions, reverberation). The authors also speculate that in the intelligibility threshold region, other quality aspects like loudness and naturalness might be so degraded that they "swamp" the intelligibility component, which could explain the poor performance of some measures.

Summary updated

BibTeX
@inproceedings{liu2006assessment,
  author = {Wei M. Liu and John S. D. Mason and Nicholas W. D. Evans and Keith A. Jellyman},
  title = {{An assessment of automatic speech recognition as speech intelligibility estimation in the context of additive noise}},
  booktitle = {Interspeech},
  year = {2006},
}
Figure