TL;DR
This paper explores how to make automatic speech recognition (ASR) systems better at predicting human speech perception, especially in noisy environments. They found that modifying ASR training to incorporate actual human confusion patterns significantly improves its ability to predict human phoneme recognition, paving the way for more human-like ASR applications.
What Problem Does This Paper Address?
Predicting how well humans understand speech, particularly in noisy conditions, is crucial for developing better hearing aids and speech coding algorithms. While existing models often rely on simulating human auditory processing, this paper investigates a different approach: using automatic speech recognition (ASR) systems to model human speech perception. Specifically, they aim to predict the intelligibility of short speech segments (phonemes) and understand how ASR systems can learn from human errors to better mimic human perception. The challenge lies in bridging the gap between ASR's objective recognition and human listeners' subjective perception, especially when dealing with the complexities of noise.
Methods
The study employs a two-pronged experimental approach:
- Experiment I: Feature Type Analysis: The researchers evaluated the suitability of standard ASR feature extraction methods for predicting human phoneme recognition. They tested several feature types, including Mel-frequency cepstral coefficients (MFCCs), Perceptual Linear Prediction (PLP) coefficients, and Rasta-PLPs, as well as spectro-temporal Gabor features. These features were fed into a Hidden Markov Model (HMM) based ASR system. The ASR system's phoneme recognition scores were then compared to human speech recognition (HSR) scores obtained from listening experiments using the same speech data.
- Experiment II: Modified ASR Training: This experiment introduced a novel training paradigm for ASR. Instead of using standard ground-truth labels, the ASR system was trained using the actual confusion patterns observed in human listening experiments. Essentially, the ASR's "target" output for a given utterance was modified to reflect how humans actually confused phonemes in that utterance. This "perceptual training" aimed to make the ASR system produce more human-like errors.
Datasets
- Oldenburg Logatome (OLLO) Corpus: This corpus, containing 150 different logatomes (phoneme triplets like CVC or VCV), was used for both ASR training and testing, as well as for the human listening experiments. The corpus includes recordings from 50 speakers with variations in speaking effort, rate, and pitch.
- HSR Test Set: A subset of the OLLO corpus was compiled for listening experiments, featuring utterances from four speakers (2 male, 2 female) in six different intrinsic variations. This resulted in 3,600 logatomes for the HSR test set.
- ASR Training/Test Sets: Two main ASR setups were used:
- Training/Test SI (Speaker-Independent): This used approximately 14,000 utterances from six speakers for training and the 3,600 utterances from the HSR test set for evaluation.
- Training/Test SD (Speaker-Dependent): This used a smaller subset of 1,800 utterances for training and 1,800 for testing, with data from the same four speakers in both sets. This setup was necessary for the modified training approach.
Key Insights and Ideas
- Feature Type Matters: Not all ASR features are equally good at predicting human perception. While standard spectral features (MFCC, PLP, Rasta-PLP) showed good correlations, spectro-temporal Gabor features exhibited different patterns and lower correlations, suggesting they capture different aspects of speech.
- PLP Features Show Promise: Perceptual Linear Prediction (PLP) features, which incorporate aspects of human auditory processing like equal loudness weighting, showed a particularly strong correlation with human recognition scores, suggesting they are well-suited for modeling human speech perception.
- SNR Mismatch Can Be Beneficial: Surprisingly, the best prediction accuracy was achieved when the Signal-to-Noise Ratio (SNR) used for ASR training and testing differed from the SNR used in human listening tests. This is attributed to the limitations of modeling extreme noise conditions in ASR.
- Learning from Human Errors is Powerful: The core innovation, "perceptual training," where ASR is trained on human confusion patterns, significantly improved the prediction of human phoneme recognition. This suggests that directly incorporating human perceptual biases into ASR training is a highly effective strategy.
Key Results
- Experiment I (Conventional Training):
- The highest correlation between ASR and HSR was achieved with PLP features at a 6 dB SNR mismatch, yielding an r-value of 0.84.
- For speaker-independent ASR, MFCC features performed best at higher SNRs, while Gabor features were superior at lower SNRs.
- Experiment II (Modified Training):
- The perceptual training approach improved the correlation between ASR and HSR scores, reaching a maximum of 0.89 (with PLP features). This is a significant improvement over the best conventional training result (0.80).
- While perceptual training led to a slight degradation in raw ASR recognition scores (8% on average), it substantially improved the correlation with human scores, indicating more human-like error patterns.
Limitations and Future Work
- Speaker Dependence: The modified training approach was primarily evaluated on speaker-dependent ASR systems due to the need for specific listener data. The authors acknowledge that for real-world applications, a speaker-independent model is desirable. This would require collecting perceptual data from a larger group of speakers.
- Partial Perceptual Modeling: While the features used (especially PLP) incorporate some aspects of human hearing, they do not represent a full perceptual model.
- Future Directions: The authors suggest combining their proposed perceptual training with existing microscopic phoneme confusion models (like those in [1, 2]) to further enhance prediction quality. They also highlight the need for speaker-independent models and larger perceptual datasets.
Summary updated