TL;DR
This paper explores using automatic speech recognition (ASR) systems to estimate human intelligibility scores for noisy speech, specifically for the Japanese Diagnostic Rhyme Test (DRT). By adapting ASR models to the specific noise conditions, they significantly improve the correlation between ASR-generated scores and human perception, paving the way for faster, automated intelligibility assessments.
What Problem Does This Paper Address?
Assessing speech intelligibility, especially in noisy environments, is crucial for evaluating communication systems. Traditional methods like the Diagnostic Rhyme Test (DRT) are highly accurate but require extensive human listening tests, making them time-consuming and costly. The challenge is to develop an automated method that can reliably predict human intelligibility scores without needing human listeners, particularly for the Japanese language and its unique phonetic structure. Previous attempts using ASR showed promise but struggled with accuracy, especially when the noise conditions for training and testing didn't match.
Methods
The core of this work involves adapting Automatic Speech Recognition (ASR) models to better mimic human intelligibility judgments. The authors employ a two-stage adaptation process:
- Speaker Adaptation: Acoustic models are first adapted to individual speakers using their recorded speech. This is done using Maximum Likelihood Linear Regression (MLLR), a standard technique for personalizing ASR models.
- Noise Adaptation: Following speaker adaptation, the models are further adapted to specific noise types (white noise, multi-talker babble, pseudo-speech noise) at defined Signal-to-Noise Ratios (SNRs) of 0 dB and 10 dB. This step is crucial for improving performance in noisy conditions.
To simulate the DRT's forced two-to-one selection process, a special "word-pair language model" is used. This model restricts the ASR's output to only one of the two words in a given minimal pair, forcing a decision based on the initial phoneme. The "correct response rate" is then calculated using a formula that accounts for chance, mirroring the DRT's scoring.
Datasets
- Japanese DRT Word List: A set of 120 words forming 60 minimal pairs, designed to test specific phonetic attributes in Japanese. This list was used for both recording speech and for the intelligibility tests.
- Corpus of Spontaneous Japanese (CSJ): Used for training initial speaker-independent acoustic models.
- Clean Speech Recordings: Recorded for four male and four female speakers for the DRT word list.
- Noisy Speech: Clean speech was artificially degraded by adding Gaussian white noise, multi-talker noise, and pseudo-speech noise at various SNRs (+10 dB, 0 dB, -10 dB, -15 dB).
Key Insights and Ideas
- Noise Adaptation is Key: While speaker adaptation improves ASR-based intelligibility scores, noise adaptation provides a significant boost, especially when the noise type and level used for adaptation match the test conditions.
- Matching Noise Levels Matters: The performance of noise-adapted models is highly sensitive to the SNR. When the adaptation SNR matches the test SNR, the ASR scores closely align with human perception.
- Degradation When SNRs Don't Match: If the test noise level is higher than the adaptation level (i.e., lower SNR), the ASR scores degrade. Conversely, if the test noise level is lower than the adaptation level (higher SNR), the ASR scores can even surpass subjective rates, suggesting a potential over-adaptation or a different kind of mismatch.
- Multi-talker Noise is Challenging (and Revealing): Noise adaptation appears particularly effective for multi-talker (babble) noise, which is more speech-like than random noise. This suggests that ASR models might better handle noise that shares characteristics with speech.
- Towards Automated Intelligibility Testing: The improved correlation between ASR-computed and subjective scores, especially with noise adaptation, suggests that ASR can serve as a valuable tool for rapid, automated intelligibility estimation, reducing the need for extensive human listening.
Key Results
- Speaker-independent models showed significantly lower correct response rates compared to subjective results, sometimes by over 50%.
- Speaker-adapted models improved the correct response rate by up to 20%, but still had a difference of up to 30% from subjective results.
- Noise-adapted models further improved correct response rates by over 20% in some cases.
- Correlation coefficients between subjective and computed correct response rates were significantly improved by adaptation. For noise-adapted models (10 dB), correlations reached over 0.91 for white and pseudo-speech noise, and 0.94 for babble noise. The 0 dB noise-adapted models achieved a correlation of 0.95 for babble noise.
- When noise levels matched (e.g., 0 dB adapted and 0 dB tested), the difference between subjective and estimated response rates was less than 10% for multi-talker noise.
Limitations and Future Work
- Sensitivity to Noise Level Mismatch: The paper highlights that performance degrades when test noise levels differ significantly from adaptation levels, especially when test SNRs are higher than adapted SNRs. This suggests that a single adaptation level might not be sufficient for all scenarios.
- Future Work: The authors propose exploring adaptation to "mixed levels" of noise. This would allow for more robust intelligibility estimation across a wider range of unknown environments without needing to sample the specific noise beforehand. They also plan to investigate adapting to mixed noise types to achieve a reasonable estimate regardless of the noise type.
Summary updated