TL;DR
This paper introduces a novel method to improve automatic speech recognition (ASR) by integrating syllable stress information directly into the decoding process. By using a rhythm model to quantify stress and reinforcing paths through stressed syllables, they achieved a relative improvement of over 2% in Word Error Rate (WER), even on foreign-accented speech.
What Problem Does This Paper Address?
Improving the accuracy of Automatic Speech Recognition (ASR) systems, particularly for large vocabulary continuous speech, is an ongoing challenge. While prosodic features like intonation and rhythm are known to influence human speech perception, their integration into ASR has been limited. Specifically, syllable stress, which makes stressed syllables acoustically more salient and perceptually distinct, has shown mixed results when incorporated into ASR systems. This paper aims to systematically investigate and demonstrate the benefits of integrating syllable stress information into the decoding stage of an ASR system, addressing the gap in effectively leveraging this linguistic feature.
Methods
The core of this work lies in a novel approach to integrate syllable stress into the ASR decoding process.
- Stress Detection: The authors employ a multiscale model of rhythm perception, the "Auditory Primal Sketch," to derive stress scores. This model quantifies the prominence of acoustic events in the speech signal, and its output, the "rhythmogram," is used to determine syllable stress. The energy of the speech signal is used to compute the rhythmogram.
- Integration into ASR: The stress information is integrated into the ASR system's search process by modifying the path scoring equation. Two strategies were explored:
- Proportional Reinforcement: Paths passing through syllables are reinforced with a value proportional to their stress score.
- Constant Reinforcement: Paths passing through stressed syllables (defined as having a stress score > 0.5) are reinforced with a constant value.
- ASR System: The experiments were conducted using "Irene," a two-pass ASR system developed at IRISA, which includes a word graph generation pass followed by a rescoring pass.
Datasets
- Development Set: Approximately 3.5 hours of French broadcast news from the ESTER2 evaluation campaign corpus were used for tuning the reinforcement parameter (R).
- Test Set: Approximately 4.5 hours of French broadcast news from the ESTER2 evaluation campaign corpus were used for evaluating the system. This corpus includes standard French and speech with strong accents (e.g., from African radio stations).
Key Insights and Ideas
- Stress as a Robust Feature: The stress information proved to be a robust feature, performing well even on foreign-accented speech, which is a significant challenge for ASR systems.
- Proportional Reinforcement Outperforms Constant: Reinforcing paths proportionally to the stress score yielded better results than a constant reinforcement for stressed syllables. This suggests that the degree of stress matters, not just its presence.
- Integration in Both Passes is Beneficial: Integrating stress information in both the first pass (word graph generation) and the second pass (rescoring) led to the best performance improvements.
- Parameter R is Flexible: The reinforcement parameter (R) could be varied over a large range without degrading performance, indicating the inherent robustness of the stress information.
Key Results
- Development Set: The proportional reinforcement strategy improved recognition when applied in the first pass (35.3% WER vs. 35.6% baseline) and even more so when applied in both passes (35.1% WER vs. 35.6% baseline). The constant reinforcement strategy showed minimal gain.
- Test Set: The proportional stress-enhancing scheme in the first step improved performance by 1.3% relative value (23.2% WER vs. 23.5% baseline). Its use in both passes brought a relative WER reduction of 2.1% (23.0% WER vs. 23.5% baseline). These improvements were statistically significant.
- Foreign Accented Speech: The stress-enhanced decoder performed well on strong-accented data, showing a relative improvement of 2.5% when stress was integrated in both passes, compared to 1.9% for non-accented speech.
- Word Graph Error Rate (WGER): Integrating stress in the first pass led to an absolute improvement of 0.6% in WGER (12.1% vs. 12.7% baseline), indicating a better word graph.
Limitations and Future Work
- Syllable Boundary Detection: Syllable boundaries were obtained by combining phone-level forced alignment with French syllabification rules. The authors suggest that using automatically segmented syllables could be a future step.
- Reinforcement Scope: Currently, the entire syllable is reinforced. Future work could explore reinforcing only the neighborhood around a rhythmogram event or using different reinforcement values within the same syllable based on acoustic likelihood.
- Alternative Stress Models: While the Auditory Primal Sketch is effective, exploring other rhythm and stress detection models could be beneficial.
- Cross-Lingual Application: While tested on accented French, further investigation into the applicability of this method to other languages would be valuable.
Summary updated