TL;DR
This paper introduces DISCO, an ASR-based Computer-Assisted Language Learning (CALL) system designed to automatically detect and provide feedback on pronunciation and grammar errors in Dutch L2 learners. The study presents the first evaluation of the complete system, highlighting its potential for improving L2 speaking skills while also identifying areas for technological and pedagogical refinement.
What Problem Does This Paper Address?
Learning a second language (L2) often involves practicing speaking skills, but receiving immediate and detailed feedback on pronunciation and grammar can be challenging. Traditional CALL systems often lack robust speech interaction capabilities, and existing speech-interactive systems may not offer granular, segment-level feedback. This gap hinders L2 learners' ability to identify and correct specific errors, particularly in connected speech. The DISCO project aims to bridge this gap by developing an ASR-powered system that can automatically assess L2 learners' spoken Dutch and provide targeted, corrective feedback.
Methods
The DISCO system employs a two-step approach for processing student utterances:
- Speech Recognition: A Hidden Markov Model (HMM)-based speech recognition system is used. Crucially, it operates with a Finite State Grammar (FSG) language model that is pre-populated with a list of predicted correct and incorrect responses for each prompt. This "forced choice" approach aims to improve accuracy by constraining the recognition possibilities.
- Error Detection:
- Syntactic and Morphological Errors: These are largely detected through the speech recognition process itself, leveraging confidence measures based on acoustic likelihood.
- Pronunciation Errors: A weighted FSG is used, which encodes the canonical phone string alongside frequently observed pronunciation errors. Transition costs are assigned to these error arcs to manage the rate of false alarms.
The system's design is based on a closed-response conversation simulation and interactive participatory drama, where learners engage with computerized agents in pre-programmed scenarios. Feedback strategies can be customized, offering immediate visual feedback or communicative recasts.
Datasets
The paper does not explicitly mention datasets used for training the ASR models. However, the evaluation utilized data collected from 23 Dutch L2 students (14 at Linguapolis, University of Antwerp, and 9 at Radboud in'to Languages, Radboud University Nijmegen). The interactions, including system prompts, student responses, and system feedback, were recorded for analysis. Expert annotations were also generated for feedback accuracy assessment.
Key Insights and Ideas
- Potential of ASR in CALL: The evaluation suggests that ASR technology holds significant promise for L2 speaking practice, with students generally appreciating the system and finding it useful for learning.
- Feedback Customization Matters: The ability to choose between immediate corrective feedback and more conversational recasts is a valuable feature, catering to different learning preferences.
- Challenges in L2 Speech: L2 learner speech presents unique difficulties for ASR, particularly with subtle phonetic distinctions and morphological variants, which can lead to recognition errors.
- Subjectivity in Error Annotation: Evaluating feedback accuracy is complex, as expert annotations can be subjective. The study highlights the need for multiple annotators and clear annotation guidelines for future evaluations.
- False Rejects vs. False Accepts: While the system might generate a high number of "false rejects" (marking correct sounds as errors), students may not always perceive this as problematic, potentially due to not noticing the feedback or assuming they made an error. This warrants further investigation.
Key Results
- User Satisfaction: Students generally reported positive experiences, finding the system easy to use, visually attractive, and the dialogues fun and realistic.
- Feedback Understanding: Most students understood the feedback provided by the system and felt they learned from it.
- Exercise Effectiveness: Students found the practice exercises useful and fun, though morphology exercises were perceived as too easy, and pronunciation exercises varied in difficulty.
- Feedback Accuracy:
- Syntax: Correct feedback was provided for 87.6% (Antwerp) and 82.4% (Nijmegen) of utterances. Correct recognition was achieved for 80.3% (Antwerp) and 77.0% (Nijmegen).
- Morphology: Correct feedback was provided for 80.2% (Antwerp) and 73.1% (Nijmegen) of utterances. Correct recognition was achieved for 71.5% (Antwerp) and 73.1% (Nijmegen).
- Pronunciation: Precision for Correct Accept (CA) was high (98.7% Antwerp, 99.4% Nijmegen), indicating few sounds incorrectly marked as correct. However, Precision for Correct Reject (CR) was lower (68.9% Antwerp, 22.7% Nijmegen), suggesting a significant number of sounds were incorrectly marked as errors (false rejects).
- Overall Program Rating: Students indicated they would use the program themselves, with average marks ranging from 9.0 to 7.2 in Belgium and 8.5 to 5.0 in the Netherlands (the lower scores in the Netherlands were attributed to technical issues).
Limitations and Future Work
- Technical Issues: The evaluation was hampered by disconnections with the speech processor in Nijmegen, affecting the system's ability to provide feedback in all cases. This limitation is acknowledged as a significant factor influencing the results.
- Subjectivity in Evaluation: The study points out the subjectivity inherent in expert annotation of errors, suggesting the need for multiple annotators and refined methodologies for future evaluations.
- Pronunciation Error Detection: The high number of false rejects in pronunciation feedback, particularly in Nijmegen, requires further investigation and optimization of the system's parameters.
- Exercise Design: Certain syntax and morphology exercises that rely on subtle acoustic differences are identified as problematic and may need to be avoided or redesigned.
- Future Plans: The authors plan a new round of evaluations with an improved system version that addresses disconnections. They also intend to optimize pronunciation error detection parameters using the collected speech material and get better insight into the accuracy of pronunciation feedback.
Summary updated