Acoustic and visual phonetic features in the mcgurk effect — an audiovisual speech illusion

Kaisa Tiippana, Mikko Tiainen, Lari Vainio, Martti VainioInterspeech 2013

TL;DR

This paper investigates how acoustic and visual speech features interact to create the McGurk effect, a fascinating audiovisual illusion. By analyzing the acoustic and visual properties of speech sounds in different vowel contexts ([a] vs. [e]) and under varying noise levels, the researchers found that the perceived speech sound is heavily influenced by the reliability and distinctiveness of these features, leading to context-dependent perceptual outcomes.

What Problem Does This Paper Address?

The McGurk effect, where visual speech articulation alters auditory perception, is a prime example of how our brains integrate information from different senses to understand speech. While the effect itself is well-documented, the specific acoustic and visual phonetic features that contribute to this integration and why they lead to different perceptual outcomes in various contexts remain less understood. This paper aims to bridge this gap by dissecting the acoustic and visual characteristics of stimuli used in the McGurk effect and correlating them with observed perceptual differences, particularly in the context of varying acoustic noise.

Methods

The study employed a combination of behavioral experiments and phonetic analysis. Participants listened to audiovisual stimuli and reported their perception. The core of the study involved analyzing the acoustic properties of speech sounds using spectrograms and formant analysis (F1 and F2) with the Praat software. Additionally, visual speech articulations were analyzed frame-by-frame from video recordings, focusing on mouth opening, tongue position, and other articulatory features. The experimental design included varying levels of acoustic noise (signal-to-noise ratios from -12 dB to +6 dB, plus a no-noise condition) and two vowel contexts ([a] and [e]). The stimuli were specifically designed McGurk pairs: an acoustic [p] paired with a visual [k] (A[p]V[k]), presented in both [a] and [e] vowel contexts.

Datasets

Not discussed in the paper. The stimuli were created by editing video recordings of two female talkers.

Key Insights and Ideas

  • Vowel Context Matters: The same acoustic [p] and visual [k] combination leads to different perceptions depending on the following vowel. In an [a] context, the visual [k] often dominates, leading to a perception of [k]. In an [e] context, the perception is more often a fusion, perceived as [t].
  • Feature Reliability Drives Perception: The distinctiveness and reliability of acoustic and visual features are crucial. When acoustic cues are degraded by noise, the visual signal's influence increases.
  • Acoustic Transitions are Key: The F2 formant transitions from the consonant to the vowel were found to be more similar between the acoustic [p] and the perceived [t] in the [e] context, explaining the fusion percept. In the [a] context, the acoustic [p] transition was more distinct from [k], favoring visual dominance.
  • Visual Distinctiveness Varies: The visual articulation of consonants was more distinct in the [a] vowel context due to a wider mouth opening and better visibility of the tongue, making the visual [k] more clearly distinguishable from other consonants.
  • Noise Amplifies Visual Influence: As acoustic noise increased, the perception shifted towards the visual component, especially in the [a] context, where the visual [k] became even more dominant.

Key Results

  • In the [a] context, the A[apa]V[aka] stimulus was predominantly heard as [aka], with the proportion of [k] responses increasing with noise level.
  • In the [e] context, the A[epe]V[eke] stimulus was mostly heard as a fusion [ete] without noise. With increasing noise, fusion [t] responses initially increased but then decreased, with a corresponding increase in [k] responses at the lowest SNRs.
  • Phonetic analysis revealed that in the [e] context, the acoustic [p] and visual [k] features were closer to those of [t], facilitating fusion. In the [a] context, visual [k] features were clear, while acoustic [p] features were less distinctive, leading to visual dominance.
  • Visual speech perception (lip-reading) was more accurate for [p] than [k] and [t] in the [a] context, and for [p] followed by [t] in the [e] context, with [k] being poorly recognized.

Limitations and Future Work

The authors acknowledge that their study focused on a specific set of stimuli (A[p]V[k]) and vowel contexts. While they found strong evidence for the role of phonetic features, they suggest that more elaborate measures of visual features could further strengthen the interpretation of the McGurk effect. Future work could explore a wider range of phonetic contexts and consonant types, as well as investigate individual differences in audiovisual integration. The study also highlights the importance of considering the physical stimulus features, which is often overlooked in McGurk effect research.

Summary updated

BibTeX
@inproceedings{tiippana2013acoustic,
  author = {Kaisa Tiippana and Mikko Tiainen and Lari Vainio and Martti Vainio},
  title = {{Acoustic and visual phonetic features in the mcgurk effect — an audiovisual speech illusion}},
  booktitle = {Interspeech},
  year = {2013},
}
Figure