Age-related changes in multisensory integration of emotions in an audiovisual face-prosody-semantics Stroop task

Yi Lin, Shumeng Ni, Yangfan LuInterspeech 2025

TL;DR

This study reveals that older adults struggle more with integrating emotional cues from different sensory channels (face, voice, and word meaning) compared to younger adults. This difficulty is particularly pronounced for nonverbal cues like voice tone and facial expressions, and it worsens when the emotional information across channels is contradictory, suggesting a role for declining cognitive control in aging.

What Problem Does This Paper Address?

As we age, our ability to communicate effectively and understand social cues can change. This includes how we process emotions, which are conveyed through multiple channels simultaneously – what someone says (semantics), how they say it (prosody), and their facial expression. While previous research has shown age-related declines in processing each of these channels individually, it's less clear how these declines interact when multiple emotional signals are presented together. Specifically, how do older adults integrate these different emotional inputs, and does this integration process differ from younger adults, especially when the signals are conflicting? This paper aims to shed light on these age-related changes in multisensory emotion integration, exploring whether deficits stem from sensory limitations or cognitive control issues.

Methods

The researchers employed a modified Stroop task, a classic cognitive paradigm, to investigate multisensory emotion integration. Participants were presented with audiovisual stimuli where happy or sad emotions were conveyed through three channels: semantic (word meaning), prosodic (vocal tone), and facial (facial expression). The core of the task involved participants selectively attending to one channel (e.g., focusing only on the facial expression) while ignoring the other two. The key manipulation was the congruence of emotions across the channels: sometimes all three channels conveyed the same emotion (congruent), and other times one channel conflicted with the other two (incongruent).

The experiment was conducted in a soundproof booth, with stimuli presented via headphones and an LCD monitor. Participants were instructed to identify the emotion of the attended channel. The study measured both accuracy (how often they got it right) and reaction time (how quickly they responded). Statistical analysis involved linear mixed-effects models to account for individual differences and the hierarchical structure of the data.

Datasets

The study utilized an existing audiovisual emotion database, previously established for research on cross-channel and cross-modal emotion perception in young adults. This database included:

  • Semantic stimuli: 32 Mandarin adjectives conveying happiness or sadness.
  • Prosodic stimuli: The same adjectives spoken by amateur actors with either a happy or sad vocal tone.
  • Facial stimuli: 32 black-and-white images of emotional faces expressing happiness or sadness.

All stimuli were pre-tested and validated to ensure high recognition rates (over 90%) and consistent emotional intensity ratings.

Key Insights and Ideas

  • Nonverbal channels are more vulnerable: Older adults show a greater decline in integrating emotions from nonverbal channels (prosody and face) compared to verbal semantics. This suggests that the challenges of aging impact the processing of auditory tone and visual expressions more significantly than the meaning of words.
  • Incongruence is the key: The age-related differences in emotion integration become much more pronounced when the emotional signals across channels are contradictory. This highlights that while older adults might manage congruent emotional information reasonably well, conflicting cues pose a substantial challenge.
  • Cognitive control plays a role: The findings, particularly the significant three-way interaction between task, congruence, and group, point towards a cognitive basis for these age-related deficits. The difficulty in ignoring irrelevant, conflicting information when attending to one channel suggests a decline in inhibitory control, a key aspect of executive function.
  • Prosody is particularly challenging: Among the nonverbal channels, prosody (vocal tone) appears to be the most affected in older adults, showing the largest age-related differences in both accuracy and reaction time, especially under incongruent conditions.

Key Results

  • Overall Accuracy: Older adults performed with significantly lower accuracy than younger adults across all tasks.
  • Task-Specific Accuracy: Age differences in accuracy were significant for facial and prosodic tasks but not for the semantic task.
  • Congruence-Specific Accuracy: While there were no significant group differences for cross-channel congruent conditions, older adults showed significantly lower accuracy in semantics-incongruent, prosody-incongruent, and face-incongruent conditions.
  • Three-Way Interaction: No significant group differences were found when information was congruent across channels. However, under incongruent conditions, older adults were less accurate in facial and prosodic tasks, with the most pronounced deficits observed in the prosodic task. No significant age differences were found in the semantic task under any congruence condition.
  • Reaction Time: Older adults were generally slower than younger adults in processing emotional information. This age-related slowing was most pronounced in the prosodic task, followed by the facial and then the semantic task.
  • No significant interaction was found between congruence and group, or among task, congruence, and group for reaction time, suggesting that while older adults are slower overall, the impact of incongruence on speed might not differ as dramatically as it does on accuracy.

Limitations and Future Work

The authors acknowledge several limitations:

  • Static vs. Dynamic Stimuli: The use of static facial images alongside dynamic auditory speech might not fully capture real-world multimodal emotion perception. Future research should incorporate more ecologically valid stimuli, such as dynamic video clips with accompanying linguistic and extralinguistic cues.
  • Precise Cognitive Correlates: While the study suggests a role for inhibitory control, it did not directly measure specific cognitive abilities. Future work should explore the precise correlations between emotion perception and measures of perceptual and cognitive functions (e.g., hearing thresholds, MoCA scores) to quantify their relative contributions.
  • Individual Variations: The study focused on group differences. Future research could investigate how individual variations in emotional speech perception within unisensory settings relate to multisensory performance, providing a more nuanced understanding of the interplay between individual and contextual factors.

The authors suggest that future work could also systematically investigate how individual variations in emotional speech perception within unisensory settings relate to their performance in multisensory contexts, offering a comprehensive picture that could inform rehabilitation and intervention strategies for aging populations.

Summary updated

BibTeX
@inproceedings{lin2025age,
  author = {Yi Lin and Shumeng Ni and Yangfan Lu},
  title = {{Age-related changes in multisensory integration of emotions in an audiovisual face-prosody-semantics Stroop task}},
  booktitle = {Interspeech},
  year = {2025},
}
Figure