TL;DR
This paper tackles the challenge of incorporating prosodic information (like pitch and duration) into Mandarin automatic speech recognition (ASR) systems. They propose a novel method for automatically labeling prosodic features on speech data and then use these labels to train prosody-dependent acoustic models, significantly improving tonal syllable recognition accuracy.
What Problem Does This Paper Address?
Automatic Speech Recognition (ASR) systems typically focus on the phonetic content of speech, often treating prosody – the rhythm, stress, and intonation of speech – as secondary or even noise. However, prosody carries crucial linguistic and paralinguistic information, such as emphasis, intent, and sentence structure, which can significantly aid in understanding spoken language. For tonal languages like Mandarin, where syllable pitch is integral to word meaning, ignoring prosody can lead to errors, especially in recognizing tones. Existing approaches to incorporate prosody often require large, manually prosodically labeled speech corpora, which are expensive and time-consuming to create. This paper aims to bridge this gap by developing methods for automatic prosodic labeling and using this to train more effective prosody-dependent ASR models for Mandarin.
Methods
The authors introduce a two-pronged approach:
- Automatic Prosody Labeling: They leverage their previously developed methods to automatically annotate continuous Mandarin speech corpora for two key prosodic features: syllable prosodic break type and stress type. This bypasses the need for manual annotation.
- Prosody-Dependent Acoustic Modeling:
- MSD-HSMM: They utilize a Multi-Space Probability Distribution Hidden Semi-Markov Model (MSD-HSMM) to model the influence of prosody on acoustic features like pitch and duration. This is an extension of the MSD-HMM, which is better suited for modeling continuous, potentially discontinuous, acoustic features.
- GMM for Duration: Gaussian Mixture Models (GMMs) are used to model prosody-dependent tonal syllable duration.
- MaxEnt for Syntax: A Maximum Entropy (MaxEnt) model is employed to incorporate syntactical prosody information.
- Combined Decoding: For decoding, they integrate the MSD-HSMM model, the prosody-dependent tonal syllable duration model, and the syntactical prosody model to adjust search path scores and improve recognition results.
A key aspect of their training strategy is to focus on prosody-dependent allophones (variants of phonemes) rather than traditional tri-phone models. This helps address the data sparsity problem that arises when trying to model all possible prosodic variations for every phonetic context.
Datasets
- Training Data: The "863" corpus, a large-vocabulary continuous speech recognition corpus for Mandarin, was used. This included data from 83 male speakers (48,373 sentences, 55.6 hours).
- Testing Data: Data from 6 male speakers (240 sentences, 17.1 minutes) from the same "863" corpus was used for testing.
- Prosodic Annotation Benchmark: For evaluating their automatic prosody labeling system, they manually annotated 120 sentences from the "863" corpus with the help of three annotators.
Key Insights and Ideas
- Automatic Prosody Labeling is Feasible: The paper demonstrates that automatic labeling of prosodic features like stress and breaks can achieve high accuracy (e.g., 95.6% for stress, 90.5% for prosodic breaks), making it a practical alternative to manual annotation.
- Prosody Significantly Improves ASR: Incorporating prosodic information, even with relatively simple labeling and modeling, leads to noticeable improvements in ASR performance, particularly for tonal syllable recognition.
- Syllable-Based Modeling is Effective: Moving from tri-phone models to syllable-based models, especially when considering prosody, helps mitigate data sparsity issues and improves recognition accuracy.
- Pitch is a Crucial Prosodic Feature: The paper highlights that pitch is a significant factor influencing speech recognition performance, and modeling it appropriately is key.
- MSD-HSMM is Suitable for Prosodic Modeling: The MSD-HSMM framework proves effective in capturing the variations in acoustic features induced by prosody.
Key Results
The proposed prosody-dependent ASR system achieved significant improvements in tonal syllable recognition:
- The Syl-Pro-MSD-HSMM_Proposed system achieved a correct rate of 76.18% for tonal syllables.
- This represents an improvement of 4.32% over the prosody-independent Syl-MSD-HMM system (71.86% correct rate).
- Compared to the baseline TriPho-HMM system (62.73% correct rate), the improvement was 13.45%.
- The error rate for tonal syllables was also reduced, dropping from 40.25% for the baseline to 30.43% for the proposed system.
Limitations and Future Work
- Data Sparsity: While their methods help, the paper acknowledges that a large speech corpus is still required for training prosody-dependent allophone models.
- Simplified Prosodic Features: The automatic labeling simplified prosodic breaks into a single type and stresses into binary (stressed/unstressed), without distinguishing finer-grained categories.
- Future Work: The authors plan to:
- Introduce prosody-dependent language models and dictionaries into the decoding process.
- Obtain Chinese characters as output.
- Label more training data to further reduce the impact of data sparsity.
Summary updated