Effects of visual speech information on native listener judgments of L2 consonant intelligibility

Saya Kawase, Yue WangInterspeech 2012

TL;DR

This study investigates how visual cues from lip movements affect the intelligibility of non-native English consonant sounds produced by Japanese speakers, as perceived by native English listeners. The findings reveal that while visual information generally aids intelligibility, it can sometimes hinder it, particularly when the speaker's articulatory production of a sound is incorrect.

What Problem Does This Paper Address?

Understanding how second language (L2) learners produce and how native speakers perceive L2 sounds is crucial for improving language learning and communication. While previous research has explored the interaction of first (L1) and second (L2) language sound inventories, and the role of auditory information in L2 speech perception, the specific impact of visual speech information on the intelligibility of L2 productions has been less explored. This paper aims to fill this gap by examining how native English listeners judge the intelligibility of English consonants produced by Japanese learners, under various audiovisual conditions. The authors are particularly interested in whether the "non-native speaker effect," where listeners increase their reliance on visual cues when processing non-native speech, leads to improved intelligibility of L2 productions, or if incorrect articulatory gestures can be detrimental when amplified by visual attention.

Methods

The study employed a controlled experimental design involving audio and video recordings of English consonant-vowel (CV) syllables.

  • Speakers: Fifteen native Japanese speakers (late L2 learners of English) and fifteen native Canadian English speakers (as a control group, carefully selected for East Asian descent to minimize perceived ethnicity bias) produced six English CV syllables: /ba/, /va/, /sa/, /θa/, /la/, and /ða/.
  • Stimuli Creation: Audio and video recordings were made of speakers reading the CV syllables. High-quality audio was captured separately from the video. The audio was normalized, and then synchronized with the video. Three stimulus conditions were created:
  • Audiovisual (AV): Both audio and synchronized video were presented.
  • Audio-only (AO): Only the audio track was presented.
  • Visual-only (VO): Only the video track (lip movements) was presented.
  • Listening Task: Thirty-one native Canadian English listeners participated. They were presented with the stimuli in a sound-treated room and asked to identify the initial consonant of each CV syllable. The task was administered using E-prime 2.0, with randomized modality conditions (AV, AO, VO). Listeners could select from a list of possible consonants or type their own response if they perceived something else.

Datasets

The study did not introduce new datasets. Instead, it created a novel dataset of recorded English CV syllables produced by native Japanese speakers and native English speakers, specifically for the purpose of this experiment. This dataset was then used to create the AV, AO, and VO stimulus conditions for evaluation.

Key Insights and Ideas

  • Visual Cues Can Be a Double-Edged Sword: While visual information often aids in understanding non-native speech, it can also lead to decreased intelligibility if the speaker's articulatory production is flawed. This challenges the assumption that visual cues are always beneficial for L2 intelligibility.
  • The "Non-Native Speaker Effect" in Action: Native listeners showed a tendency to rely more on visual cues when processing non-native (Japanese-produced) English sounds, consistent with previous findings.
  • L1 Influence Extends to Visual Perception: Similar to auditory perception, the influence of the speaker's native language (Japanese) was observed in the visual perception of English consonants. Sounds that are shared between Japanese and English were generally perceived better, even visually.
  • Specific Consonants Show Differential Visual Effects: The study identified specific consonants where visual information was particularly facilitative (e.g., /b/, /v/, /θ/) and others where it was less so or even detrimental (e.g., /l/).

Key Results

  • Overall AV Benefit: In the AV condition, Japanese-produced consonants were perceived as significantly more intelligible than in the AO condition, indicating a general facilitative effect of visual speech.
  • Detrimental Effect for /l/: The Japanese production of the consonant /l/ was perceived as less intelligible in the AV condition compared to the AO condition. This suggests that incorrect articulatory configurations for /l/ by Japanese speakers may have been amplified by visual attention, leading to decreased accuracy.
  • Visual-Only Performance: In the VO condition, the intelligibility of non-native phonemes like /θ/, /v/, and /l/ produced by Japanese speakers was generally lower than in the AO condition, suggesting that auditory information is crucial for these sounds.
  • Consonant-Specific Differences:
    • For Japanese productions, /b/, /s/, and /l/ were significantly more intelligible than /v/, /θ/, and /l/ in the AO condition.
  • Visual information significantly improved the perception of /b/, /v/, and /θ/ in AV compared to AO.
  • However, /s/ and /l/ showed higher intelligibility in AO than VO, though AV still provided a boost for /s/.
  • The intelligibility of /l/ in AV was significantly lower than in AO and VO, highlighting its unique susceptibility to visual interference.

Limitations and Future Work

  • Limited Stimuli: The study used only six consonants and one vowel (/a/). Future research could explore a wider range of phonemes and syllable structures.
  • Specific L2 Learner Group: The study focused on Japanese learners of English. Investigating other L1 backgrounds would provide broader insights into L2 speech perception.
  • Articulatory Accuracy: While the study infers incorrect articulatory configurations for /l/, direct articulatory analysis was not performed. Future work could correlate articulatory precision with perceptual intelligibility.
  • Nature of Visual Weighting: The paper suggests increased visual weighting for non-native speech but calls for further studies on the precise nature of these visual modality influences and the interaction between auditory and visual input.
  • Listener-Based Assessments: The findings suggest that listener-based assessments of L2 production should consider the impact of visual cues, as they can be both facilitative and detrimental.

Summary updated

BibTeX
@inproceedings{kawase2012effects,
  author = {Saya Kawase and Yue Wang},
  title = {{Effects of visual speech information on native listener judgments of L2 consonant intelligibility}},
  booktitle = {Interspeech},
  year = {2012},
}
Figure