A Japanese dialogue-based CALL system with mispronunciation and grammar error detection

Oh Pyo Kweon, Akinori Ito, Motoyuki Suzuki, Shozo MakinoInterspeech 2004

TL;DR

This paper introduces a novel dialogue-based Computer-Assisted Language Learning (CALL) system for Japanese that goes beyond traditional pronunciation drills. It enables learners to practice "real conversations" by allowing them to compose their own utterances, with the system providing feedback on both pronunciation and grammar, and dynamically adjusting the dialogue flow.

What Problem Does This Paper Address?

Learning a new language effectively requires more than just memorizing vocabulary and practicing pronunciation. Traditional CALL systems often place learners in a passive role, offering limited opportunities for them to actively compose their own sentences or engage in dynamic, conversational practice. Furthermore, many existing systems focus solely on pronunciation exercises, neglecting the crucial aspects of grammar and sentence composition needed for true communication fluency. This paper addresses the need for a CALL system that simulates real-world conversations, allowing learners to practice speaking, composing, and receiving feedback on both pronunciation and grammar in an interactive, dialogue-driven environment.

Methods

The proposed CALL system is designed to facilitate "real conversation" practice. Its core innovation lies in allowing learners to actively compose their own utterances in response to system prompts, rather than selecting from pre-defined options. The system then employs speech recognition to analyze these utterances, checking for both pronunciation and grammatical errors.

For pronunciation error detection, the system utilizes mispronunciation rules derived from previous analyses. When a learner speaks, the system generates sentences with potential mispronunciations and compares the likelihood and duration of the learner's speech against these generated sentences and a native speaker's utterance. A normalized likelihood difference is calculated, and a score is assigned to categorize the pronunciation quality.

Grammar problem detection is approached in two ways:

  1. Grammatical and lexical errors: The system defines specific dialogue situations (e.g., souvenir task, hotel reservation) to predict likely learner utterances. Even if learners make errors like "I want to buying" instead of "I want to buy," the system can recognize and point them out by extending its vocabulary and grammar dictionary.
  2. Out-of-grammar utterances: To reduce instances where learners produce sentences that don't conform to the system's grammar, a pre-exercise phase is introduced. This involves vocabulary, grammar, and typical conversation examples before the main dialogue exercise.

The dialogue control component is designed to be dynamic, meaning the conversation path changes based on the learner's responses, mimicking natural conversation flow.

Datasets

  • Japanese speech DB: Used to construct baseline phoneme models.
  • Foreigners' speech DB [11]: Used to train the baseline models, creating "trained models" specifically for non-native speakers. This dataset includes speech from Korean learners of Japanese.
  • Interview data: Collected from two Korean speakers with varying levels of Japanese proficiency (1 year and 1.6 years of study) for evaluating the effectiveness of pre-exercises on reducing out-of-grammar utterances and grammatical errors.

Key Insights and Ideas

  • Active Composition is Key: Learners benefit significantly from actively composing their own sentences rather than relying on pre-set options. This is crucial for developing genuine communication skills.
  • Integrated Feedback: Combining pronunciation and grammar feedback within a single dialogue system is more effective for comprehensive language learning than isolated exercises.
  • Pre-Exercise Efficacy: Introducing a pre-exercise phase (vocabulary, grammar, sample conversations) demonstrably reduces the number of out-of-grammar utterances and grammatical errors made by learners during actual conversation practice.
  • Tailored Models Improve Recognition: Training speech recognition models on non-native speaker data (specifically, Korean learners of Japanese) significantly improves recognition accuracy compared to using generic baseline models.

Key Results

  • Pronunciation Evaluation: The system's pronunciation evaluation scores showed a correlation coefficient of 0.72 with native speaker evaluations, indicating good reliability.
  • Grammar Improvement with Pre-Exercise:
    • The percentage of out-of-grammar utterances decreased from 23.5% to 11.5% when pre-exercises were used.
  • The number of grammatical errors per utterance decreased from 0.26 to 0.19 with pre-exercises.
  • Recognition Rate Improvement:
    • For male speakers, the correct/accuracy rate increased from 57.97%/10.23% to 66%/13.37% after training with foreigner speech data.
  • For female speakers, the rate increased from 61.78%/15.10% to 66%/19.96%.
  • Word correct rate for interview-derived sentences increased from 79.7% to 81.1% using trained models.

Limitations and Future Work

  • Limited Interview Data: The authors acknowledge that the interview data collected for evaluating pre-exercise effectiveness was small. They plan to collect more data to conduct a more thorough analysis of grammatical and lexical errors, particularly considering the influence of learners' overall language abilities.
  • Grammar Error Analysis: A more comprehensive analysis of grammar errors, especially those related to language proficiency, is needed.
  • Ambiguous Pronunciations: While training on ambiguous pronunciations improved recognition rates, further work is needed to fully address the challenges posed by slow speaking speeds and ambiguous pronunciations common in foreign language learners.

Summary updated

BibTeX
@inproceedings{kweon2004japanese,
  author = {Oh Pyo Kweon and Akinori Ito and Motoyuki Suzuki and Shozo Makino},
  title = {{A Japanese dialogue-based CALL system with mispronunciation and grammar error detection}},
  booktitle = {Interspeech},
  year = {2004},
}
Figure