In the field of speech processing, speech enhancement (SE) is often employed as a front-end to improve the performance of back-end automatic speech recognition (ASR) systems in noisy environments. Traditionally, the effectiveness of SE algorithms is evaluated using instrumental measures like PESQ (Perceptual Evaluation of Speech Quality) or STOI (Short-Time Objective Intelligibility). However, these metrics were designed to model human auditory perception and often fail to correlate well with the Word Error Rate (WER) of ASR systems. This paper introduces the Cross-Entropy-Guided (CEG) measure, a novel metric specifically designed to predict ASR performance by leveraging the internal states of the ASR system itself.
The Need for ASR-Centric Metrics
The primary goal of speech enhancement for ASR is to reduce WER. Measuring WER directly is computationally expensive and requires manual transcriptions, making it impractical for rapid algorithm development or real-time optimization. While PESQ and STOI are convenient, they focus on speech quality and intelligibility for human listeners. A speech enhancement algorithm might improve these perceptual qualities while simultaneously introducing artifacts or distortions that degrade the ASR system's ability to recognize phonemes. This discrepancy creates a significant gap between "human-friendly" and "machine-friendly" audio.
The Cross-Entropy-Guided (CEG) Framework
The proposed CEG measure bridges this gap by utilizing the ASR system’s acoustic model to evaluate the impact of speech enhancement. Unlike traditional black-box metrics, CEG looks at the high-level internal representations generated by the ASR system.
As illustrated in the figure above, the CEG calculation involves three main stages:
- Feature Extraction: Both clean and enhanced speech signals are converted into low-level acoustic features (e.g., MFCCs or FBANK features).
- Nonlinear Mapping: These features are passed through the ASR system's acoustic model (typically a deep neural network). Instead of final text output, the measure extracts the state posterior probabilities—the network's internal confidence scores for various speech units (phonemes) at each time frame.
- Measure Calculation: The CEG is computed as the cross-entropy between the posterior probabilities of the clean speech and those of the enhanced speech. By comparing these internal "guesses," CEG quantifies how much the enhancement process has deviated from the ideal acoustic representation expected by the ASR back-end.
Experimental Evaluation
The researchers validated the CEG measure using the CHiME-4 benchmark, which includes diverse noise types (Cafe, Street, Bus, Pedestrian) and various signal-to-noise ratios (SNRs) from -5dB to 20dB. The evaluation spanned multiple acoustic model architectures (DNN, DNNsMBR, DCNN) and language models (3-gram, 5-gram, RNN) to ensure the metric's robustness across different ASR configurations.
Key Results and Insights
The experimental results demonstrate that CEG is a far more reliable predictor of ASR performance than traditional metrics.
As shown in the scatter plots, CEG exhibits a significantly higher Pearson correlation with WER compared to PESQ and STOI. In multi-condition training modes, CEG achieved a correlation of approximately 0.74, whereas PESQ and STOI lagged behind at 0.55 and 0.63, respectively. This superior performance was consistent across different noise types and ASR back-end architectures.
A critical insight from this study is that CEG can identify instances of "over-processing." Some SE algorithms might yield high PESQ scores by aggressively removing noise, but in doing so, they "smear" the acoustic cues essential for the ASR model. CEG detects this as a mismatch in the internal probability distributions, correctly predicting a rise in WER where traditional metrics might predict an improvement.
Conclusion and Future Potential
The CEG measure provides a powerful tool for developing ASR-friendly speech enhancement front-ends. Its primary advantage lies in its ability to reflect the specific needs of the machine back-end rather than human perception. Furthermore, because CEG is based on a differentiable cross-entropy function, it holds significant potential as a loss function for training deep learning-based SE models directly to minimize ASR error.
Future work will involve extending the CEG framework to modern end-to-end (E2E) ASR architectures, where internal representations differ from the state posteriors used in hybrid ANN-HMM systems.
Summary updated