Role of prosody in automatic modality recognition of bangla speech

Anal Warsi, Tulika Basu, Debasis MazumdarInterspeech 2012

TL;DR

This paper explores how the "music" of speech (prosody) can automatically identify the intent behind a Bangla sentence (statement, question, or command). By analyzing pitch and energy patterns, they developed a system that achieves over 96% accuracy, a significant step for improving speech translation and human-like speech synthesis.

What Problem Does This Paper Address?

Automatic Speech Recognition (ASR) systems are crucial for speech-to-speech translation, but they often struggle to capture the nuances of human intent. A sentence like "You're going to the store" can be a statement, a question, or even a command depending on how it's spoken. This ambiguity is particularly challenging in languages like Bangla, where the same words can be used for different sentence modalities. The paper aims to bridge this gap by automatically recognizing sentence modality using prosodic features, which are the non-lexical elements of speech like pitch, rhythm, and intonation. This capability is vital for more accurate translations and for generating more natural-sounding synthetic speech.

Methods

The researchers developed a system to classify Bangla utterances into three modalities: Statement (S), Question (Q), and Command (C).

  • Feature Extraction: They extracted a comprehensive set of 680 features from recorded speech samples, focusing on pitch (fundamental frequency) and energy contours. These features included statistical measures like minimum, maximum, mean, quartiles, and slopes of the pitch and energy over the entire utterance and specifically for the last prosodic word.
  • Feature Selection: To identify the most discriminative features, they employed several feature selection algorithms (Linear Forward Selection, Best First, Genetic Search, Exhaustive Search, and Greedy Search). This process reduced the initial set of 30 features to a more manageable subset of 8 highly informative features.
  • Classification: Three well-known machine learning algorithms were used to classify the utterances: Decision Tree J48, Support Vector Machine (SVM), and k-Nearest Neighbor (k-NN). These classifiers were trained and tested on both the full feature set and the reduced subset.

Datasets

The study utilized a newly created Bangla speech corpus.

  • Bangla Speech Corpus: This corpus consists of 680 utterances recorded by 10 native speakers (5 male, 5 female) of Standard Colloquial Bangla (SCB), aged 25-35. The sentences were sourced from Bangla storybooks. Recordings were digitized at 16 kHz with 16-bit accuracy in a studio environment. A listening test involving 30 native Bangla speakers was conducted to ensure the clarity and unambiguous categorization of each utterance, resulting in a final dataset with 229 Statements, 235 Questions, and 216 Commands.

Key Insights and Ideas

  • Prosody is Key: The study strongly suggests that prosodic features, particularly pitch and intensity, are highly effective in distinguishing between statements, questions, and commands in Bangla.
  • Terminal Prosody Matters: Features derived from the end of an utterance, especially the last prosodic word, were found to be particularly discriminative. This aligns with the typical Subject-Object-Verb sentence structure in Bangla, where verbs carrying the mood often appear at the end.
  • Feature Reduction Boosts Performance: Using a carefully selected subset of features significantly improved the classification accuracy for all tested algorithms compared to using the full feature set. This highlights the importance of identifying and utilizing only the most relevant information.
  • k-NN Shines: The k-Nearest Neighbor algorithm, when combined with the reduced feature set, demonstrated superior performance, achieving the highest overall accuracy.

Key Results

  • Highest Accuracy: The k-NN classifier, using the reduced subset of 8 features, achieved a remarkable 96.08% global accuracy in classifying Bangla sentence modalities.
  • Full Feature Set Performance: With the full set of 30 features, k-NN achieved 89.21% accuracy, J48 achieved 86.27%, and SVM achieved 84.31%.
  • Reduced Feature Set Performance: Using the reduced feature set, k-NN reached 96.08%, J48 reached 90.68%, and SVM reached 88.23%.
  • Command Class Improvement: The reduced feature set led to a notable improvement in the recognition of the Command class, which had previously underperformed with the full feature set.

Limitations and Future Work

  • Limitations: The paper notes that the Command class was still the most challenging to classify, although performance improved with the reduced feature set. The authors also mention that the listening test was crucial for ensuring data quality, implying that real-world noisy conditions might pose further challenges.
  • Future Work: The authors suggest that further research could explore other prosodic features and potentially incorporate acoustic features beyond pitch and energy. Expanding the dataset to include more speakers and a wider variety of sentence structures could also enhance the robustness of the system.

Summary updated

BibTeX
@inproceedings{warsi2012role,
  author = {Anal Warsi and Tulika Basu and Debasis Mazumdar},
  title = {{Role of prosody in automatic modality recognition of bangla speech}},
  booktitle = {Interspeech},
  year = {2012},
}
Figure