Abstract
Automated speech scoring systems typically require extensive human-labeled datasets for each target language, posing a significant challenge for low-resource languages. This paper introduces a language-independent framework that leverages self-supervised multilingual representations (wav2vec 2.0 XLSR) and expert-driven feature engineering to assess pronunciation and fluency without language-specific training data. Evaluated on a dataset of 50,000 audio files across five languages, the system achieves Pearson correlations exceeding 0.63 for unseen languages. These results suggest that multilingual bootstrapping can effectively bridge the gap for low-resource speech assessment, reaching performance levels comparable to models trained on language-specific data.
Introduction
The proliferation of automated speech assessment in language learning and recruitment has highlighted a critical bottleneck: the dependency on large-scale, human-annotated datasets. Developing a "Language-Specific" model for a new language requires thousands of expert-rated audio clips, a resource-intensive process that excludes many low-resource languages. This research addresses the "data scarcity" problem by proposing a universal scorer capable of cross-lingual generalization—learning the characteristics of "good" speech from high-resource languages and applying them to entirely new, unseen linguistic contexts.
Methodology
The proposed framework integrates deep learning with traditional linguistic feature extraction to ensure both performance and interpretability.
- Multilingual Foundation: The system utilizes wav2vec 2.0 XLSR, pre-trained on 53 languages, to extract robust acoustic representations. Two variants are employed: one fine-tuned for phoneme recognition and another for Automatic Speech Recognition (ASR).
- Alignment and Segmentation: To determine the temporal boundaries of speech units, a Connectionist Temporal Classification (CTC) based algorithm aligns expected text with audio, providing precise timestamps and phoneme-level probability scores.
- Feature Engineering: Rather than relying solely on end-to-end deep learning, the authors extract specific features categorized into:
- Force-Alignment Based: Including speech rate, pause duration, and phoneme posterior probabilities.
- ASR-Based: Including Phoneme Error Rate (PER) and Word Error Rate (WER).
- Cross-Lingual Training: The "Language-Independent" model is trained using a leave-one-language-out strategy, where the model is trained on a subset of languages and evaluated on a completely unseen target language.
- Normalization: To account for cross-linguistic variations in baseline speech metrics (e.g., average speech rate), unlabeled data from the target language is used to normalize feature scales.
Experimental Setup
The study utilized a proprietary dataset from the SVAR spoken language assessment platform, comprising nearly 50,000 audio files from 2,500 job-seekers. The dataset covers five languages: English, French, Spanish, German, and Portuguese. Ground truth was established using a 5-point rubric (ranging from "extremely uneven" to "completely fluent") with ratings provided by three human experts per response, maintaining a high inter-rater agreement ($r > 0.8$).
Results and Discussion
The performance of the Language-Independent model was compared against Language-Specific (trained on the target) and Language-Combined (trained on all available data) models.
- High Cross-Lingual Correlation: The Language-Independent model achieved Pearson correlations ($r$) of over 0.65 for pronunciation and 0.63 for fluency across all unseen languages.
- Typological Benefits: Performance significantly improved when typologically similar languages were included in the training set. For instance, including Spanish data improved Portuguese fluency correlation by 30%.
- Generalization Patterns: Pronunciation features (e.g., phoneme likelihoods) demonstrated higher universality than fluency features (e.g., pause patterns), which appeared more culturally and linguistically sensitive.
- German Performance: Notably, for German, the Language-Independent model showed a slight improvement in pronunciation accuracy over the monolingual model, suggesting that multilingual training can capture a more generalized understanding of phonetic quality.
Conclusion
This work demonstrates the feasibility of high-accuracy speech scoring for unseen languages. While Language-Specific models remain the gold standard, the Language-Independent approach provides a powerful "bootstrap" for immediate deployment in low-resource scenarios. Future research will focus on extending this framework to tonal and non-European languages to further validate its universal applicability and investigate potential demographic biases.
Summary updated