TL;DR
This paper introduces a novel, language-independent feature set for automatically evaluating the naturalness of prosody in spoken language. By focusing on acoustic features of voiced segments rather than relying on speech recognition or specific language models, their method achieves comparable performance to language-specific approaches and human evaluators, opening doors for more universal language learning tools.
What Problem Does This Paper Address?
Learning a foreign language involves mastering not just vocabulary and grammar, but also prosody – the rhythm, stress, and intonation that convey meaning and naturalness. While crucial, prosody is often neglected in language learning due to the difficulty of providing personalized feedback. Existing automatic systems often focus on single-word pronunciation or require language-specific models, limiting their scalability. This paper tackles the challenge of creating a language-independent system that can automatically assess the naturalness of a speaker's prosody, making it applicable to any language without extensive retraining.
Methods
The core innovation lies in a "text-independent" feature extraction method. Instead of relying on speech recognition to align text with speech, this approach directly analyzes acoustic properties of the speech signal.
- Prosodic Base-Features: The system extracts fundamental frequency (F0), energy, and identifies voiced and unvoiced speech segments.
- Segmentation and Feature Computation: Voiced segments are merged if separated by less than 50ms, and F0 contours are interpolated for robustness. For each segment (and "context segments" of two adjacent merged segments), a rich set of 187 features is computed. These include standard prosodic measures like F0, energy, and duration, but also novel spectral descriptions derived from FFT coefficients of the F0 and energy contours.
- Text-Independent Features: To create a single feature vector per segment, the mean value of each of the 187 segment features is computed.
- Naturalness Prediction: Support Vector Regression (SVR) is employed to predict the naturalness score based on these feature vectors. SVR is chosen for its ability to model complex, non-linear relationships and its robustness.
- Feature Selection: Correlation-based Feature Subset (CFS) selection is used to identify the most informative features from the large set, aiming to improve performance and efficiency.
A contrasting "word-based" approach is also presented, which does rely on speech recognition output for time alignment and phoneme information to extract prosodic features. This serves as a baseline for comparison.
Datasets
The study uses a custom dataset comprising recordings of the same text passage from "The Little Prince" read in both German and Japanese.
- Participants: 26 individuals (13 German, 13 Japanese) who were fluent in both languages.
- Recordings: Each participant read both the German and Japanese versions of the text.
- Annotation: The recordings were independently evaluated by two native speakers of German and two native speakers of Japanese, rating prosody naturalness on a scale of 1 to 5. The mean of these ratings served as the ground truth for automatic evaluation.
Key Insights and Ideas
- Prosody is Universal: The paper strongly suggests that acoustic features of prosody can be analyzed independently of the specific language, challenging the notion that prosody evaluation systems must be language-specific.
- Voiced Segments are Key: Focusing on voiced segments, rather than words or phonemes, provides a robust basis for extracting prosodic information across languages.
- Spectral Features Add Value: Incorporating spectral descriptions of F0 and energy contours seems to capture important aspects of naturalness that simpler features might miss.
- SVR is Effective: Support Vector Regression proves to be a capable tool for mapping complex prosodic features to human perception of naturalness.
Key Results
- German Data:
- The text-independent SVR system achieved a correlation of 0.88 with human ratings for German, comparable to the inter-rater correlation of 0.88.
- The word-based system performed slightly better, achieving 0.92 after feature selection, but this still showed no significant difference from the text-independent approach.
- Japanese Data:
- The text-independent SVR system achieved a correlation of 0.83 with human ratings for Japanese (after feature selection), with an inter-rater correlation of 0.92.
- Crucially, the performance difference between the text-independent system and human raters was not statistically significant for Japanese either.
- Feature Selection: For the word-based system on German, feature selection improved performance from 0.89 to 0.92. For the text-independent system, feature selection on Japanese improved performance from 0.76 to 0.83.
Limitations and Future Work
- Language-Specific Nuances: While the text-independent features performed well, the correlation for Japanese (0.83) was lower than for German (0.88). The authors suggest this might be because the features were originally designed and evaluated with German speech, and some features may not be as meaningful for Japanese. Future work could involve optimizing feature sets for specific language families or further refining the universal feature set.
- Data Size: The dataset, while carefully collected and annotated, is relatively small (26 speakers). Expanding the dataset with more speakers and potentially more diverse text passages would strengthen the findings.
- Beyond Naturalness: The current system focuses solely on prosodic naturalness. Future work could extend this to evaluate other aspects of prosody, such as clarity, expressiveness, or emotional content.
Summary updated