Acoustics-guided evaluation (AGE): a new measure for estimating performance of speech enhancement algorithms for robust ASR

Li Chai, Jun Du, Chin-Hui LeearXiv 2018 · arXiv

Introduction

Speech Enhancement (SE) algorithms are critical for robust Automatic Speech Recognition (ASR) in noisy environments. Traditionally, SE performance is evaluated using Word Error Rate (WER). However, WER calculation is computationally expensive and requires manual transcriptions, making it unsuitable for rapid iterative development. Common proxy metrics like PESQ and STOI, designed for human perception, often fail to correlate with ASR performance because they do not account for the specific acoustic cues utilized by ASR backends.

This paper introduces Acoustics-Guided Evaluation (AGE), a novel metric that estimates ASR performance by leveraging high-level representations from a pre-trained acoustic model. AGE provides a more accurate prediction of WER without the need for manual transcriptions or full decoding.

Methodology: The AGE Framework

The AGE framework shifts the evaluation from raw waveforms to the internal representations of an ASR system's Acoustic Model (AM). The process involves three primary stages:

  1. Feature Extraction: Standard low-level features (e.g., MFCCs or Filterbanks) are extracted from both clean reference speech and degraded (noisy or enhanced) speech.
  2. High-level Representation: These features are processed by a pre-trained neural network AM. AGE specifically extracts the State Posterior Probabilities (SPP), which represent the model's frame-level confidence across different phonemes.
  3. Metric Calculation: The final AGE value is determined by calculating the cross-entropy between the SPP distributions of the clean and degraded signals.

By utilizing cross-entropy at the SPP level, AGE quantifies how much the enhancement process or noise has distorted the acoustic information critical for ASR classification.

Experimental Evaluation

The researchers validated AGE using the CHiME-4 1-channel task, which features real-world noise environments (Cafe, Street Junction, Public Transport, and Pedestrian Area). The dataset included simulated data mixed at Signal-to-Noise Ratios (SNRs) from -5dB to 20dB.

To test robustness, the metric was evaluated across various ASR architectures:

  • Acoustic Models: DNN, DCNN, and DNNsMBR.
  • Language Models: 3-gram, 5-gram, and RNN.

Results and Key Insights

Experimental results demonstrate that AGE consistently outperforms traditional metrics in predicting WER.

Superior Correlation

AGE achieved significantly higher Pearson correlation coefficients (ρ) compared to Entropy, PESQ, and STOI. As shown in the comparative analysis, AGE maintains a high correlation (reaching ~0.74 in multi-condition training) across different acoustic and language model architectures, whereas traditional perceptual metrics lag behind.

Pearson correlation coefficients for various metrics across different Acoustic and Language Models. AGE (dark red) consistently shows the highest correlation with WER.

Robustness and Predictability

The primary advantage of AGE is its backend-awareness. Unlike perceptual metrics that are backend-invariant, AGE incorporates the specific "knowledge" of the ASR system. This allows it to detect subtle artifacts introduced by SE algorithms that might be inaudible to humans but detrimental to ASR accuracy. Furthermore, the relationship between AGE and WER is more monotonic and predictable than that of PESQ or STOI.

Conclusion and Future Directions

AGE represents a significant advancement in ASR-oriented speech enhancement evaluation. By aligning the evaluation metric with the ASR backend, researchers can more accurately predict performance gains during the development cycle.

Future research will focus on:

  • Reference-free AGE: Developing versions that do not require a clean reference signal.
  • End-to-End Architectures: Extending the framework to modern end-to-end ASR systems.
  • Direct Optimization: Utilizing AGE as a loss function to train SE models specifically for ASR performance rather than just signal reconstruction.

Summary updated

Abstract

One challenging problem of robust automatic speech recognition (ASR) is how to measure the goodness of a speech enhancement algorithm (SEA) without calculating the word error rate (WER) due to the high costs of manual transcriptions, language modeling and decoding process. Traditional measures like PESQ and STOI for evaluating the speech quality and intelligibility were verified to have relatively low correlations with WER. In this study, a novel acoustics-guided evaluation (AGE) measure is proposed for estimating performance of SEAs for robust ASR. AGE consists of three consecutive steps, namely the low-level representations via the feature extraction, high-level representations via the nonlinear mapping with the acoustic model (AM), and the final AGE calculation between the representations of clean speech and degraded speech. Specifically, state posterior probabilities from neural network based AM are adopted for the high-level representations and the cross-entropy criterion is used to calculate AGE. Experiments demonstrate AGE could yield consistently highest correlations with WER and give the most accurate estimation of ASR performance compared with PESQ, STOI, and acoustic confidence measure using Entropy. Potentially, AGE could be adopted to guide the parameter optimization of deep learning based SEAs to further improve the recognition performance.
BibTeX
@misc{chai2018acoustics,
  author = {Li Chai and Jun Du and Chin-Hui Lee},
  title = {{Acoustics-guided evaluation (AGE): a new measure for estimating performance of speech enhancement algorithms for robust ASR}},
  year = {2018},
  eprint = {1811.11517},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1811.11517},
}
Figure