Real-time visualization of English pronunciation on an IPA chart based on articulatory feature extraction

Yurie Iribe, Takurou Mori, Kouichi Katsurada, Goh Kawai, Tsuneo NittaInterspeech 2012

TL;DR

This paper introduces a novel system for real-time visualization of English pronunciation on an International Phonetic Alphabet (IPA) chart. By extracting articulatory features (AFs) from speech and mapping them to a visual chart, the system aims to provide learners with intuitive feedback on their pronunciation, helping them correct errors more effectively than traditional methods.

What Problem Does This Paper Address?

Learning a foreign language, particularly pronunciation, can be challenging. While Computer Assisted Pronunciation Technology (CAPT) systems exist, many struggle to provide learners with clear, actionable feedback. Existing methods often rely on acoustic features like F1-F2 planes, which require a deep understanding of phonetics to interpret. Learners often lack the intuitive grasp of how their tongue, lips, and jaw movements translate into specific sounds. This paper addresses the need for a system that visually represents pronunciation in a way that directly links to articulatory gestures, making it easier for learners to understand and correct their mistakes.

Methods

The core of the system is an Articulatory Feature (AF) extraction process. This involves:

  • Acoustic Feature Extraction: Input speech is processed using Fast Fourier Transform (FFT) and Mel-scaled band-pass filters to extract local features (LFs). These LFs capture spectral variations.
  • Multi-Layer Neural Networks (MLNs): The extracted LFs are fed into a series of MLNs. The first MLN maps acoustic features to discrete AFs, while a second MLN refines these AFs by considering phoneme boundaries. Subsequent stages incorporate inhibition/enhancement functionalities and decorrelate context vectors to produce robust AFs.
  • Articulatory Feature Representation: AFs are represented as a set of binary (+/-) values corresponding to distinctive phonetic features (e.g., "voiced," "bilabial," "front vowel"). The system generates AF sequences for phonemes.
  • Real-time Visualization: The extracted AF sequences are converted into 2D x-y coordinates. These coordinates are then plotted in real-time (every 10 ms) onto specialized "pronunciation maps" based on the IPA chart.
  • Vowel Map: Plots vowel height (open-close) on the y-axis and vowel backness (front-back) on the x-axis, using a trapezoidal shape to represent the IPA vowel chart.
  • Consonant Map: Plots consonant features, often pairing phonemes that Japanese speakers commonly confuse (e.g., [b] vs. [v]). The axes represent specific differentiating AFs.

Datasets

The paper utilizes several datasets for training and evaluation:

  • TIMIT: Used for training and testing the AF extractor for English phonemes.
  • Training: 2600 sentences from 325 male English native speakers.
  • Testing: 896 sentences from 112 male English native speakers.
  • CSJ (Corpus of Spontaneous Japanese): Used for training and testing the AF extractor for Japanese phonemes.
  • Training: 22 hours of speech from 92 male Japanese native speakers.
  • Testing: 2.5 hours of speech from 10 male Japanese native speakers.
  • "The North Wind and The Sun" sentences: Used for testing the plotting accuracy of the system with Japanese speakers producing English speech. 15 sentences from 15 male Japanese speakers.

Key Insights and Ideas

  • Bridging the Acoustic-Articulatory Gap: The most significant insight is the successful translation of complex acoustic speech signals into interpretable articulatory features and their subsequent visualization on an IPA chart. This directly addresses the learner's need to understand how to move their mouth and tongue.
  • Intuitive Feedback: The use of visual "pronunciation maps" provides immediate, intuitive feedback. Learners can see their pronunciation represented as a dot, allowing them to compare it to target phoneme positions and adjust their articulation accordingly.
  • Focus on Common Errors: The consonant map's design, pairing phonemes frequently confused by Japanese speakers, highlights a targeted approach to error correction.
  • Detailed Articulation Representation: The system's ability to consider not just the maximum AF value but also the next highest value allows for a more nuanced representation of articulation, especially for vowels.

Key Results

  • High AF Extraction Accuracy: The system achieved high correct rates for articulatory features, averaging 93% for English phonemes and approximately 96% for unique Japanese phonemes.
  • Promising Plotting Accuracy:
    • Vowel pronunciation map: Results showed that phonemes with higher AF extraction accuracy were plotted closer to their correct positions.
  • Consonant pronunciation map: Achieved over 50% accuracy for most phonemes, with exceptions for [v], [θ], and [ð].
  • Qualitative Findings: The system could visualize typical English pronunciation errors made by Japanese speakers, such as substituting Japanese vowels for English ones (e.g., [æ], [A], [a] plotted around [a_J]).

Limitations and Future Work

  • AF Extractor Improvement: The authors acknowledge that the accuracy of the pronunciation maps is strongly correlated with the accuracy of AF extraction. They specifically identify the need to improve the extraction of features like "dental" and "labiodental" for consonants, and "back" for vowels.
  • Expanding Feature Set: Future work includes developing functions to indicate AFs for "round" and "tense" vowels, which are crucial for accurate vowel pronunciation.
  • Educational Effect Evaluation: The authors plan to conduct experiments to evaluate the educational effectiveness of the system within CAPT applications.

Summary updated

BibTeX
@inproceedings{iribe2012real,
  author = {Yurie Iribe and Takurou Mori and Kouichi Katsurada and Goh Kawai and Tsuneo Nitta},
  title = {{Real-time visualization of English pronunciation on an IPA chart based on articulatory feature extraction}},
  booktitle = {Interspeech},
  year = {2012},
}
Figure