Electrophysiological representations of multivariate human emotion experience

Jin Liu, Xin Hu, Xinke Shen, Sen Song, Dan Zhang

TL;DR

This paper argues that human emotions are complex and often co-occur, moving beyond traditional single-emotion studies. Using EEG and Representational Similarity Analysis (RSA), they found that brain activity in Alpha and Beta bands, particularly in frontal and parietal-occipital regions, reflects this multivariate emotional experience. This work highlights the importance of a holistic approach to understanding the neural basis of emotions.

What Problem Does This Paper Address?

Most neuroscience research on emotions has focused on understanding the neural basis of single, discrete emotions (e.g., happiness, sadness). However, in real life, emotions are rarely experienced in isolation; they often blend and co-occur. This "multivariate emotion experience" presents a challenge for traditional univariate approaches, which may not fully capture the complexity of human affective states. This paper aims to bridge this gap by investigating how the brain represents these complex, co-occurring emotions.

Methods

The core of this study is the application of inter-situation Representational Similarity Analysis (RSA). Here's how it works:

  1. Multivariate Emotion Experience: Participants rated their experience of 8 different emotion categories (anger, disgust, fear, sadness, amusement, inspiration, joy, tenderness) after watching video clips. These ratings formed an 8-dimensional "multivariate emotion experience" vector for each situation (video).
  2. EEG Representation: Electroencephalography (EEG) was used to record brain activity. The researchers extracted Power Spectral Density (PSD) features from 30 EEG channels across five frequency bands (Delta, Theta, Alpha, Beta, Gamma). This resulted in a high-dimensional EEG representation for each situation.
  3. RSA Core: RSA works by comparing the similarity structure of representations. In this "inter-situation" approach, the researchers calculated the pairwise Euclidean distances (dissimilarities) between the emotion experience vectors for all pairs of videos. They did the same for the EEG PSD features.
  4. Correlation: The key step is to correlate these two dissimilarity matrices. A significant correlation indicates that the patterns of similarity in brain activity (EEG) mirror the patterns of similarity in reported emotional experience.
  5. Reliability and Specificity: To ensure their findings were robust, the researchers conducted reliability analyses by varying the number of videos and participants. They also performed ablation analyses, removing one emotion category at a time from the ratings, to test the specificity of the EEG representations for the multivariate experience.

Datasets

The study utilized an EEG dataset collected from 78 participants who watched 28 emotional video clips. These clips were specifically chosen to elicit a range of emotions, including four negative (anger, disgust, fear, sadness), four positive (amusement, inspiration, joy, tenderness), and neutral emotions. The video durations varied, with a mean of 63 seconds.

Key Insights and Ideas

  • Emotions are messy: The behavioral data confirmed that even videos intended to elicit a single emotion often evoked a blend of multiple emotions, supporting the study's premise of multivariate emotion experience.
  • Alpha and Beta bands are key: The most significant correlations between EEG activity and reported emotion were found in the Alpha and Beta frequency bands. This suggests these brain oscillations play a crucial role in processing complex emotional states.
  • Frontal and Parietal-Occipital regions matter: These significant correlations were primarily localized to the frontal and parietal-occipital brain regions, indicating these areas are central to representing multivariate emotions.
  • Multivariate is more than the sum of its parts: The study found that the EEG representations for multivariate emotion experience were distinct from those for individual, univariate emotions, suggesting that the brain processes blended emotions differently than isolated ones.
  • Reliability is key: The findings remained largely consistent even when reducing the number of videos or participants, suggesting the identified EEG representations are robust.

Key Results

  • 70 out of 150 PSD features showed significant correlations with multivariate emotion experience ratings.
  • Significant correlations were predominantly found in the Alpha (29 channels) and Beta (21 channels) bands.
  • Representative channels with the highest correlation values included O2 in the Alpha band (r = 0.3286) and PO4 in the Beta band (r = 0.3386).
  • Reliability analyses showed that the EEG representations were largely preserved with more than 18 videos or 40 participants.
  • Ablation analyses indicated that the multivariate EEG representations were relatively stable and consistent, with a limited impact from excluding individual emotion variates.

Limitations and Future Work

  • Spatial Resolution: The authors acknowledge that EEG has relatively low spatial resolution, limiting their ability to pinpoint specific brain regions. They suggest that neuroimaging techniques like fMRI, with higher spatial resolution, would be beneficial for further investigation.
  • Stimulus Variety: While video clips are common, the generalizability of findings could be improved by exploring other emotion elicitation methods (e.g., music, recall tasks).
  • Complexity of Stimuli: The study focused on basic emotions. Future work could explore more complex, high-level emotions like aesthetic emotions to gain a more comprehensive understanding of multivariate emotion representation.
  • Individual vs. Shared Representations: While this study identified general representations, future research could delve deeper into exploring the interplay between shared and individual-specific neural representations of emotions.

Summary updated

Abstract

<b>ABSTRACT</b>Despite the fact that human daily emotions are co-occurring by nature, most neuroscience studies have primarily adopted a univariate approach to identify the neural representation of emotion (emotion experience within a single emotion category) without adequate consideration of the co-occurrence of different emotions (emotion experience across different emotion categories simultaneously). To investigate the neural representations of multivariate emotion experience, this study employed the inter-situation representational similarity analysis (RSA) method. Researchers used an EEG dataset of 78 participants who watched 28 video clips and rated their experience on eight emotion categories. The EEG-based electrophysiological representation was extracted as the power spectral density (PSD) feature per channel in the five frequency bands. The inter-situation RSA method revealed significant correlations between the multivariate emotion experience ratings and PSD features in the Alpha and Beta bands, primarily over the frontal and parietal-occipital brain regions. The study found the identified EEG representations to be reliable with sufficient situations and participants. Moreover, through a series of ablation analyses, the inter-situation RSA further demonstrated the stability and specificity of the EEG representations for multivariate emotion experience. These findings highlight the importance of adopting a multivariate perspective for a comprehensive understanding of the neural representation of human emotion experience.
BibTeX
@inproceedings{liu2023electrophysiological,
  author = {Jin Liu and Xin Hu and Xinke Shen and Sen Song and Dan Zhang},
  title = {{Electrophysiological representations of multivariate human emotion experience}},
  booktitle = {Cognition},
  year = {2023},
  doi = {10.1080/02699931.2023.2297272},
}
Figure