TL;DR
This paper introduces a novel, language-independent system for automatically assessing word stress in second language learners. By comparing the prosodic contours (pitch and energy) of a student's speech to a reference, the system achieves an Equal Error Rate (EER) comparable to phonetic quality assessment, making it a promising tool for computer-aided language learning.
What Problem Does This Paper Address?
Learning to pronounce a new language often involves mastering not just individual sounds (phonetics) but also the rhythm and melody of speech (prosody). Word stress, the emphasis placed on certain syllables within a word, is a crucial prosodic feature that can significantly impact intelligibility and naturalness. Misplaced stress can lead to confusion or misunderstanding, as illustrated by the English examples "object" (noun vs. verb) and "white house" vs. "Whitehouse." While existing computer-aided language learning (CALL) systems have made strides in evaluating pronunciation and fluency, automatically assessing word stress, especially in a language-independent manner, remains a challenging and underexplored area. This paper aims to fill that gap by developing a robust system for automatic word stress assessment.
Methods
The proposed system employs a "top-to-bottom" scheme that compares a student's utterance to a reference utterance. The core idea is to assess the trend similarity of prosodic features, specifically the fundamental frequency (F0) and energy contours.
Here's a breakdown of the key steps:
- Feature Extraction:
- F0 Contour: The speech signal is sampled, filtered, and then the fundamental frequency is extracted and converted to a semitone scale. This contour is then smoothed, normalized, and discontinuities are filled.
- Energy Contour: The intensity of the speech signal is estimated for each frame.
- MFCCs: Mel-frequency cepstral coefficients (MFCCs) are computed, along with their first and second derivatives, to capture spectral characteristics.
- Alignment: A crucial step is aligning the reference and student utterances. Instead of using traditional speech recognition metrics, the paper uses the Dynamic Time Warping (DTW) algorithm on the MFCC sequences to find the optimal frame-by-frame correspondence between the two utterances. This alignment is then used to compare the prosodic contours.
- Trend Similarity Assessment: The system compares the falling-rising trend of the F0 and energy contours. A trend similarity measure (TS) is calculated, which is a weighted combination of the similarity between the reference and student energy contours and the similarity between their F0 contours. The similarity itself is based on the correlation between the aligned contours.
- Decision: A final decision is made based on whether the calculated trend similarity exceeds a predefined threshold, indicating whether the student's stress pattern matches the reference.
The novelty lies in using DTW for alignment to enable frame-by-frame comparison of prosodic trends, rather than relying on global DTW distances or requiring text transcriptions for syllable-level analysis. The system is designed to be text-independent.
Datasets
The study utilized a custom database recorded at the Speech Processing and Transmission Laboratory (LPTV) at Universidad de Chile. This database consists of:
- Content: Twelve specific English words ("machine," "alone," "under," etc.), each uttered with all possible stress variants.
- Speakers: A mix of expert English speakers (a professor and advanced students) and non-expert speakers with intermediate English proficiency.
- Recording Conditions: Three microphones (Shure PG58 and two desktop PC mics) were used, and recordings were made in an office environment.
- Total Utterances: Approximately 864 utterances were recorded in total.
For experimental evaluation, a subset of this data was used, including phonetically segmented and hand-labeled utterances to assess DTW alignment accuracy under speaker and pronunciation quality mismatch conditions.
Key Insights and Ideas
- Prosodic Trends Matter: The paper strongly suggests that focusing on the trends of pitch and energy contours, rather than just their absolute values, is key to assessing word stress. This captures the dynamic nature of speech.
- DTW for Alignment, Not Recognition: Using DTW for aligning speech segments is standard, but its application here is novel. It's used to enable a robust frame-by-frame comparison of prosodic features, even when speakers or pronunciation quality differ, which is a significant departure from its typical use in speech recognition.
- Language Independence: The system's design, relying on universal prosodic features and a text-independent approach, holds promise for assessing stress in any language, a significant advantage over systems tied to specific language orthography or phonology.
- Combined Features are Better: The results indicate that both F0 and energy contours contribute valuable information for stress assessment. Optimal performance was achieved when both were used in conjunction, rather than relying on just one.
Key Results
- Overall Accuracy: The proposed system achieved an Equal Error Rate (EER) of 21.5%. This is highlighted as being comparable to the error rates observed in phonetic quality evaluation schemes, suggesting practical applicability.
- DTW Robustness: Experiments demonstrated that the DTW alignment is robust to speaker mismatch (alignment error increased by only 1.36%) and pronunciation quality mismatch (alignment error increased by only 0.62%). This is a critical finding, as these mismatches often degrade the performance of speech systems.
- Optimal Feature Weighting: The best performance (lowest EER) was achieved with a specific weighting (α = 0.49) for the F0 and energy contour similarity, indicating that a balanced contribution from both features is optimal.
Limitations and Future Work
- Limitations:
- The system was tested primarily with English language learners. While designed to be language-independent, further validation across more languages is needed.
- The paper acknowledges that incorporating duration information could further improve accuracy, but notes that this is not straightforward within the current DTW alignment framework.
- Future Work:
- Investigate techniques to improve robustness to noise.
- Integrate the proposed stress assessment system with phonetic quality and duration evaluation for a more comprehensive pronunciation assessment tool.
- Extend the system's applicability to a wider range of languages.
Summary updated