From categories to dimensions: spatio-temporal dynamics of the cerebral representations of emotion in voice

Bruno L. Giordano, Whiting Whiting, Nikolaus Kriegeskorte, Sonja A. Kotz, Pascal Belin, Joachim GroßbioRxiv 2018 · DOI

TL;DR

This study investigates how the human brain processes emotions in voices, specifically whether it uses discrete categories (like "happy" or "sad") or continuous dimensions (like "valence" and "arousal"). Using fMRI and MEG, the researchers found that the brain initially categorizes emotions very quickly, and then refines these perceptions using dimensional information at later stages. This work provides a crucial temporal dimension to our understanding of how emotions are represented in the brain.

What Problem Does This Paper Address?

For decades, researchers have debated whether emotions are best understood as distinct, discrete categories or as points along continuous dimensions. This debate extends to how the brain represents these emotions. While some theories propose specialized brain modules for each basic emotion, others suggest that emotions are better described by a few underlying dimensions like valence (pleasantness) and arousal (intensity). Previous neuroimaging studies have provided evidence for both perspectives, but it's been unclear how these two representational systems interact and evolve over time. This paper directly tackles this by comparing categorical and dimensional models of emotion processing in the voice, using high-resolution brain imaging to track the dynamics of neural activity.

Methods

The researchers employed a multi-modal approach combining behavioral assessments with neuroimaging techniques:

  • Behavioral Assessments: Participants listened to a set of voice stimuli that were systematically morphed to represent a range of emotions. They then rated the dissimilarity between these stimuli and categorized the emotions, as well as rated their valence and arousal.
  • Neuroimaging:
    • fMRI (functional Magnetic Resonance Imaging): Used to measure brain activity with high spatial resolution, identifying where in the brain emotions are processed.
  • MEG (Magnetoencephalography): Used to measure brain activity with high temporal resolution, revealing when different processing stages occur.
  • Representational Similarity Analysis (RSA): This is the core analytical technique. RSA compares the patterns of brain activity (represented as "representational dissimilarity matrices" or RDMs) to models of how the stimuli are perceived (also represented as RDMs). By comparing brain RDMs to categorical and dimensional models of emotion, the researchers could determine which model best explained the neural data at different points in time and in different brain regions.

Datasets

The study used a custom-created dataset of nonverbal emotional vocalizations called the "Montreal Affective Voices" (MAV) database. This database was extended and manipulated through morphing to create a rich set of stimuli that systematically varied in emotional content, allowing for fine-grained analysis of both categorical and dimensional aspects of emotion perception.

Key Insights and Ideas

  • The Brain is a "Categorizer First, Dimension Refiner Later": The most striking finding is the temporal separation of categorical and dimensional processing. The brain appears to rapidly categorize emotions in voices, and only later uses dimensional information to refine these perceptions.
  • Auditory Cortex as an Early Categorization Hub: The auditory cortex, particularly early sensory areas, shows strong associations with the categorical model of emotion very early on (as early as 77ms).
  • Subcortical Structures for Later Refinement: Structures like the amygdala and insula become involved in processing dimensional aspects (arousal and valence) at later latencies (e.g., >500ms).
  • Interplay of Models: It's not an "either/or" situation. Both categorical and dimensional models are important for understanding emotion processing in voices, but they operate on different timescales and in different brain regions.

Key Results

  • Behavioral data showed that while both categories and dimensions influenced perception, categories had a stronger influence.
  • MEG data revealed that the auditory cortex selectively encoded emotion categories from early latencies (77ms).
  • At later latencies, activity in these areas was characterized by selective encoding of arousal and, to a lesser extent, valence.
  • The left inferior frontal gyrus showed preferential representation of discrete emotions as early as 117ms.
  • The amygdala showed strong associations with arousal at latencies within 237-757ms, while valence was more weakly associated and represented later in the insula (>700ms).
  • Direct comparison of models showed that the categorical model uniquely explained significantly more MEG RDM variance at early latencies, while the dimensional model explained more variance at later latencies in the auditory cortex.

Limitations and Future Work

  • Generalizability: While the study used a well-controlled stimulus set, future work could explore how these findings generalize to more naturalistic and complex emotional vocalizations.
  • Individual Differences: The study focused on group-level analysis. Investigating individual differences in how people categorize and dimensionally represent emotions could provide further insights.
  • Other Modalities: The study focused on vocal stimuli. Exploring how these categorical and dimensional representations unfold for emotions expressed in other modalities (e.g., facial expressions, body language) would be valuable.
  • Clinical Implications: Understanding the temporal dynamics of emotion processing could have implications for understanding and treating conditions where emotion recognition is impaired.

Summary updated

Abstract

Abstract Whether the human brain represents emotional stimuli as discrete categories or continuous dimensions is still widely debated. Here we directly contrasted the power of categorical and dimensional models at explaining behavior and cerebral activity in the context of perceived emotion in the voice. We combined functional magnetic resonance imaging (fMRI) and magneto-encephalography (MEG) to measure with high spatiotemporal precision the dynamics of cerebral activity in participants who listened to voice stimuli expressing a range of emotions. The participants also provided a detailed perceptual assessment of the stimuli. By using representational similarity analysis (RSA), we show that the participants’ perceptual representation of the stimuli was initially dominated by discrete categories and an early (<200ms) cerebral response. These responses showed significant associations between brain activity and the categorical model in the auditory cortex starting as early as 77ms. Furthermore, we observed strong associations between the arousal and valence dimensions and activity in several cortical and subcortical areas at later latencies (>500ms). Our results thus show that both categorical and dimensional models account for patterns of cerebral responses to emotions in voices but with a different timeline and detail as to how these patterns evolve from discrete categories to progressively refined continuous dimensions. One Sentence Summary: Emotions expressed in the voice are instantly categorized in cortical processing and their distinct qualities are refined dimensionally only later on.
BibTeX
@misc{giordano2018from,
  author = {Bruno L. Giordano and Whiting Whiting and Nikolaus Kriegeskorte and Sonja A. Kotz and Pascal Belin and Joachim Groß},
  title = {{From categories to dimensions: spatio-temporal dynamics of the cerebral representations of emotion in voice}},
  year = {2018},
  doi = {10.1101/265843},
  howpublished = {bioRxiv},
}
Figure