Weighted RSA: an improved framework on the perception of audio-visual affective speech in left insula and superior temporal gyrus

Junhai Xu, Haibin Dong, Fei Guo, Zeyu Wang, Jianguo Wei, Jianwu DangbioRxiv 2020 · DOI

Abstract

Being able to accurately perceive the emotion expressed by the facial or verbal expression from others is critical to successful social interaction. However, only few studies examined the multimodal interactions on speech emotion, and there is no consistence in studies on the speech emotion perception. It remains unclear, how the speech emotion of different valence is perceived on the multimodal stimuli by our human brain. In this paper, we conducted a functional magnetic resonance imaging (fMRI) study with an event-related design, using dynamic facial expressions and emotional speech stimuli to express different emotions, in order to explore the perception mechanism of speech emotion in audio-visual modality. The representational similarity analysis (RSA), whole-brain searchlight analysis, and conjunction analysis of emotion were used to interpret the representation of speech emotion in different aspects. Significantly, a weighted RSA approach was creatively proposed to evaluate the contribution of each candidate model to the best fitted model. The results of weighted RSA indicated that the fitted models were superior to all candidate models and the weights could be used to explain the representation of ROIs. The bilateral amygdala has been shown to be associated with the processing of both positive and negative emotions except neutral emotion. It is indicated that the left posterior insula and the left anterior superior temporal gyrus (STG) play important roles in the perception of multimodal speech emotion.

Introduction

Accurately perceiving emotions through facial expressions and verbal cues is fundamental to social interaction. Despite its importance, the neural mechanisms underlying multimodal integration of affective speech remain poorly understood. Most existing research has focused on unimodal stimuli (visual or auditory alone) or used simplified audio-visual stimuli like music, failing to capture the complexity of real-world speech which combines semantic and prosodic information.

This study addresses two primary gaps: identifying the specific brain regions responsible for integrating simultaneous facial and vocal emotional signals, and developing a modeling framework that accounts for the multifaceted nature of neural emotional representation. While standard Representational Similarity Analysis (RSA) typically tests brain activity against single theoretical models, this research introduces "Weighted RSA" to evaluate how a combination of emotional models best fits neural data.

Methods

The study employed functional Magnetic Resonance Imaging (fMRI) on 25 healthy volunteers to observe brain activity during the perception of affective speech. The experimental paradigm utilized stimuli from the Geneva Multimodal Emotion Portrayals (GEMEP) dataset, consisting of 40 video clips of professional actors expressing four emotions: Anger, Sadness, Neutral, and Joy. These were presented in three modalities: visual-only, auditory-only, and combined audio-visual.

Experimental paradigm of the fMRI study, illustrating the sequence of baseline, fixation, stimulus presentation, and emotional judgment.

The technical innovation of this work is the Weighted RSA framework. Unlike standard RSA, which measures the dissimilarity between brain activity patterns and a single model, Weighted RSA uses a weighted linear combination of eight candidate models (Anger, Sad, Neutral, Joy, Negative, Visual, Audio, and Audio-Visual). Through an iterative process to minimize mean square error, the authors quantified the specific contribution of each emotion type to the activity in various Regions of Interest (ROIs). This was complemented by whole-brain searchlight and conjunction analyses to identify areas consistently active across emotional tasks.

Results

The Weighted RSA approach demonstrated that "fitted" composite models significantly outperformed any single-category model in explaining neural data (p < 0.05, FDR corrected). This suggests that neural representations of emotion are inherently complex and multifaceted.

Key regional findings include:

  • Multimodal Hubs: Conjunction analysis identified the left posterior insula (MNI: -57, -36, 12) and left anterior superior temporal gyrus (STG) (MNI: -39, -21, -3) as primary hubs for perceiving multimodal speech emotion across all valences.
  • The Amygdala's Role: The bilateral amygdala showed high sensitivity to both positive (Joy) and negative (Anger/Sadness) emotions but remained relatively unresponsive to neutral stimuli. In the left amygdala, the "Anger" model (weight: 0.404) and "Joy" model (weight: 0.208) were the primary contributors, supporting its role as a salience detector for high-arousal emotional states.
  • Hemispheric Bias: Contrary to some theories of right-hemisphere dominance for emotion, this study found a distinct left-side bias for multimodal speech perception. This is likely due to the subconscious engagement of language-processing centers triggered by speech stimuli.

Discussion and Conclusion

The findings highlight the critical roles of the left posterior insula and left anterior STG in integrating audio-visual affective cues. The failure of any ROI to correlate significantly with the "Sadness" model alone suggests high individual variability in the perception of low-arousal negative emotions compared to high-arousal states like joy or anger.

While the study provides a robust framework for understanding multimodal integration, it focused primarily on ventral brain regions. Future research should investigate dorsal regions, such as the hippocampus and anterior cingulate gyrus, which may also contribute to integration. Additionally, using pseudo-speech could help isolate pure prosodic processing from semantic interference, further clarifying the left-hemisphere's role in affective perception.

Summary updated

Abstract

Abstract Being able to accurately perceive the emotion expressed by the facial or verbal expression from others is critical to successful social interaction. However, only few studies examined the multimodal interactions on speech emotion, and there is no consistence in studies on the speech emotion perception. It remains unclear, how the speech emotion of different valence is perceived on the multimodal stimuli by our human brain. In this paper, we conducted a functional magnetic resonance imaging (fMRI) study with an event-related design, using dynamic facial expressions and emotional speech stimuli to express different emotions, in order to explore the perception mechanism of speech emotion in audio-visual modality. The representational similarity analysis (RSA), whole-brain searchlight analysis, and conjunction analysis of emotion were used to interpret the representation of speech emotion in different aspects. Significantly, a weighted RSA approach was creatively proposed to evaluate the contribution of each candidate model to the best fitted model. The results of weighted RSA indicated that the fitted models were superior to all candidate models and the weights could be used to explain the representation of ROIs. The bilateral amygdala has been shown to be associated with the processing of both positive and negative emotions except neutral emotion. It is indicated that the left posterior insula and the left anterior superior temporal gyrus (STG) play important roles in the perception of multimodal speech emotion.
BibTeX
@misc{xu2020weighted,
  author = {Junhai Xu and Haibin Dong and Fei Guo and Zeyu Wang and Jianguo Wei and Jianwu Dang},
  title = {{Weighted RSA: an improved framework on the perception of audio-visual affective speech in left insula and superior temporal gyrus}},
  year = {2020},
  doi = {10.1101/2020.08.31.276485},
  howpublished = {bioRxiv},
}
Figure