TL;DR
This paper introduces a novel "duration confidence" measure for speech recognition systems, leveraging the timing of phonetic units within words. By combining this duration-based confidence with traditional acoustic confidence, the researchers significantly improve the overall accuracy of speech recognition systems, particularly in challenging conditions.
What Problem Does This Paper Address?
In speech recognition, it's crucial to know how confident the system is about its recognized words. This "confidence measure" (CM) is vital for various applications like detecting out-of-vocabulary words, improving training, or reordering hypotheses. While acoustic information (how well the sound matches the model) is a primary source for CMs, this paper explores the underutilized potential of phonetic unit durations. The challenge is that simply looking at individual phone durations isn't enough; their timing within the context of a word and speaking rate variations needs to be considered to create a robust confidence measure.
Methods
The core of this work is a new duration model and a corresponding confidence measure.
- Duration Model: The authors propose a "Duration Model Tree" (DMT) that estimates the expected duration of phonetic units. The key innovation here is that the DMT models durations based on a large context, considering not just immediate neighbors but also broader phonetic environments within the training data. This allows for more accurate context-dependent duration predictions.
- Duration Confidence Measure: This measure quantifies the difference between the actual observed durations of phonetic units in a recognized word and the expected durations predicted by the DMT. To handle variations in speaking rate, durations are normalized. The Jeffries-Matusita distance is used to measure the discrepancy between the actual and expected duration distributions.
- Combination: The proposed duration confidence is then combined with a standard "acoustic confidence" measure (based on posterior probabilities from a hybrid HMM-ANN system) using simple linear interpolation.
Datasets
The experiments were conducted on isolated word recognition tasks across seven European languages (Catalan, English, French, German, Italian, Portuguese, Spanish).
- Training Data: Large corpora of hundreds of hours of speech from thousands of speakers were used for training both acoustic and duration models.
- Test Data: SpeechDat project databases were used, featuring application words from telephone channels. To increase the challenge and test robustness, the system vocabulary was extended to 2000 words, and noise (car, babble, restaurant, street) was added to the original signals with Signal-to-Noise Ratios between 10-20 dB.
Key Insights and Ideas
- Duration is Informative: Phonetic unit durations, especially when modeled with large contexts, provide valuable information for confidence scoring that is complementary to acoustic information.
- Context is King for Durations: The duration of a phonetic unit is heavily influenced by its surrounding sounds. The DMT's ability to capture large contexts is crucial for accurate duration modeling.
- Combining Strengths: While duration confidence alone might not outperform acoustic confidence, its combination with acoustic confidence leads to significant improvements, especially in noisy conditions.
- Speaking Rate Normalization is Essential: Accounting for variations in speaking rate is critical for making duration-based comparisons meaningful.
Key Results
- Improved System Performance: Combining acoustic and duration confidence measures led to a 6% to 10% relative improvement in Equal Error Rate (EER) on isolated word recognition tasks across several languages.
- Robustness in Noise: The combined confidence measure showed particular benefits in noisy conditions, where acoustic information alone can be less reliable.
- Duration Confidence Performance: In clean conditions, the duration confidence measure's EER was often higher than the acoustic confidence. However, even in these cases, the combination still yielded improvements.
- Out-of-Vocabulary (OOV) Performance: The duration confidence measure proved useful for rejecting OOV words, demonstrating its complementary nature.
Limitations and Future Work
- Insertion Errors: The paper notes that while insertion errors (extra words recognized) can lead to incorrect duration profiles, deleted words (words missed by the recognizer) are not effectively detected by the current duration model, as they are often recognized as silence.
- Future Work: The authors suggest that this duration-based confidence measure can be extended to connected word recognition. Further research could explore more sophisticated methods for handling deleted words and potentially integrating other linguistic features into the confidence scoring mechanism.
Summary updated