SPIRE-fluent: A Self-Learning App for Tutoring Oral Fluency to Second Language English Learners

Chiranjeevi Yarra, Aparna Srinivasan, Sravani Gottimukkala, Prasanta Kumar GhoshInterspeech 2019

TL;DR

This paper introduces SPIRE-fluent, a novel Android application designed to help second language English learners improve their oral fluency. The app provides automatic, detailed feedback on pronunciation quality and pause placement, offering a self-learning tool that addresses a gap in current language learning technologies.

What Problem Does This Paper Address?

Oral fluency in a second language is crucial for effective communication and is often characterized by correct pronunciation and appropriate use of pauses. While computer-aided language learning (CALL) has seen advancements, there's a notable lack of mobile applications that offer automatic, detailed feedback specifically on oral fluency, particularly concerning pronunciation and pause analysis. Existing tools often fall short in providing the granular, real-time feedback necessary for learners to self-correct and improve. This paper aims to fill that gap by presenting SPIRE-fluent, an app that provides learners with immediate, actionable insights into their speaking patterns.

Methods

SPIRE-fluent employs a client-server architecture. The front-end, developed using Android SDK, provides the user interface for learners. This includes a recording UI for practicing sentences, a preliminary feedback UI showing word and sentence-level scores, and a detailed feedback UI offering in-depth analysis. The back-end, running on a server, processes the recorded audio.

Key techniques include:

  • Forced Alignment: Used to estimate phoneme transcriptions and their boundaries from the learner's audio.
  • Syllabification: A tool (P2TK syllabifier) is used to convert phoneme transcriptions into syllable transcriptions, enabling syllable boundary detection.
  • Pause Detection: Features are computed for each syllable to classify them into three categories: boundary syllable with a short pause, boundary syllable with a long pause, or not a boundary syllable. This classification is based on work by Ananthakrishnan et al. [2].
  • Score Computation:
    • Word-level Score: Calculated using the Goodness of Pronunciation (GOP) measure for each phoneme within a word, normalized across the phoneme set. The formula involves a tanh function applied to the difference between expert and learner GOPs.
  • Sentence-level Score: An average of word-level scores and scores related to pause correctness. A penalty is applied if non-word-final syllables are incorrectly identified as boundary syllables. Otherwise, the pause score for a word is based on the probability of its final syllable being a boundary syllable, matching the expert's pattern.

Datasets

The paper does not explicitly mention specific datasets used for training models. However, it states that the stimuli used in the SPIRE-fluent app are taken from material used for spoken English training [6]. The expert audio was recorded from a voice-over artist proficient in British English.

Key Insights and Ideas

  • Holistic Fluency Feedback: The app goes beyond simple pronunciation scoring by integrating pause analysis, recognizing that both are critical components of oral fluency.
  • Granular Feedback: Providing both word-level and sentence-level scores, along with visual cues for syllables and pauses, allows learners to pinpoint specific areas of weakness.
  • Self-Learning Empowerment: By offering automatic, detailed feedback, SPIRE-fluent empowers learners to practice and correct themselves independently, a significant advantage for those without easy access to human tutors.
  • Visualizing Pauses: The use of colored strips to represent short and long pauses, and highlighting boundary syllables, makes abstract concepts like prosody more tangible for learners.

Key Results

The paper focuses on the architecture and functionality of the SPIRE-fluent app rather than presenting quantitative performance metrics like accuracy scores on a benchmark dataset. The key "result" is the successful development and demonstration of the app, which provides:

  • Word-level pronunciation quality scores.
  • Sentence-level fluency scores.
  • Identification and visualization of syllables and pauses (short and long) in both learner and expert speech.
  • Highlighting of syllables that are incorrectly identified as boundary syllables.

The demonstration section confirms the implementation of forced alignment, syllabification, pause detection, and score computation using specific tools and technologies.

Limitations and Future Work

  • Effectiveness Measurement: The authors acknowledge that further investigations are required to measure the effectiveness of the proposed app.
  • Feedback Parameter Sufficiency: The sufficiency of the current feedback parameters for self-learning needs to be analyzed.
  • No Quantitative Performance Metrics: The paper does not provide objective performance metrics (e.g., accuracy of pronunciation scoring, pause detection accuracy) on any dataset, making it difficult to compare its performance against other potential systems.

Future work would likely involve user studies to validate the app's effectiveness and potentially refine the feedback mechanisms based on learner outcomes.

Summary updated

BibTeX
@inproceedings{yarra2019spire,
  author = {Chiranjeevi Yarra and Aparna Srinivasan and Sravani Gottimukkala and Prasanta Kumar Ghosh},
  title = {{SPIRE-fluent: A Self-Learning App for Tutoring Oral Fluency to Second Language English Learners}},
  booktitle = {Interspeech},
  year = {2019},
}
Figure