A link between individual differences in multisensory speech perception and eye movements

Demet Gurler, Nathan Doyle, Edgar Y. Walker, John F. Magnotti, Michael S. BeauchampAttention 2015 · DOI

Abstract

The McGurk effect—a multisensory illusion where mismatched auditory and visual speech signals fuse into a new percept—exhibits significant individual variability. While some individuals are highly susceptible, others are nearly immune. This study investigates whether differences in gaze behavior explain this variability. We found that "strong perceivers" of the illusion spend significantly more time fixating on the speaker's mouth compared to "weak perceivers." This increased mouth-looking time provides a higher-fidelity visual signal, reducing sensory noise and facilitating multisensory integration.

Introduction

The McGurk effect is a classic demonstration of how the brain integrates auditory and visual information during speech perception. When presented with the audio "ba" and a video of a mouth saying "ga," many listeners perceive "da." However, the frequency of this fusion varies wildly across the population. This research addresses the gap between perceptual variability (the frequency of fusion) and behavioral variability (where individuals look during speech). We hypothesized that fixating on the mouth provides a higher-quality visual signal, making the brain more likely to integrate it with auditory input.

Methods

We monitored the gaze of 40 participants using infrared eye tracking while they viewed congruent and incongruent (McGurk) audiovisual speech stimuli. Participants reported the syllable they heard after each clip.

To analyze the underlying mechanisms, we employed the Noisy Encoding of Disparity (NED) model. This computational approach characterizes participants based on two parameters: Sensory Noise (the precision of internal signal assessment) and Disparity Threshold (the level of audiovisual mismatch tolerated before integration fails).

Figure 1: Distribution of McGurk perception across 40 subjects (a) and average gaze heatmaps for strong (b) and weak (c) perceivers.

Key Results

Our analysis revealed a significant positive correlation ($r = 0.34, P = 0.03$) between the percentage of time spent looking at the speaker's mouth and the frequency of experiencing the McGurk effect. Specifically, for every $2\%$ increase in mouth-looking time, the likelihood of perceiving the illusion increased by approximately $10\%$.

Figure 2: Correlation between the percentage of time spent looking at the mouth and the frequency of the McGurk percept.

"Strong perceivers" (those experiencing the illusion $>50\%$ of the time) spent an average of $41\%$ of the time fixating on the mouth, whereas "weak perceivers" spent only $23\%$. The NED model confirmed that mouth-fixaters exhibited lower sensory noise and higher disparity thresholds, suggesting that foveal vision of the mouth provides a "trustworthy" signal that the brain's integration centers, such as the Superior Temporal Sulcus, are more likely to utilize.

Discussion

These findings suggest that gaze behavior acts as a signal quality filter. A control experiment demonstrated that weak perceivers can still accurately identify mouth movements when forced to look at the eyes, proving that the information is visible but likely discounted due to the higher noise associated with peripheral vision.

While gaze behavior explains roughly $12\%$ of the variance in McGurk perception, other factors like spatial attention likely contribute to the remaining variability. Future work should explore the stability of these gaze patterns over time and whether training individuals to focus on the mouth can enhance multisensory integration, particularly in noisy environments like the "cocktail party" scenario.

Summary updated

BibTeX
@inproceedings{gurler2015link,
  author = {Demet Gurler and Nathan Doyle and Edgar Y. Walker and John F. Magnotti and Michael S. Beauchamp},
  title = {{A link between individual differences in multisensory speech perception and eye movements}},
  booktitle = {Attention},
  year = {2015},
  doi = {10.3758/s13414-014-0821-1},
}
Figure