Space-by-time manifold representation of dynamic facial expressions for emotion categorization

Ioannis Delis, Chaona Chen, Rachael E. Jack, Oliver Garrod, Stefano Panzeri, Philippe G. SchynsJournal of Vision 2016 · DOI

TL;DR

This paper introduces a novel "space-by-time manifold decomposition" to understand how our brains represent dynamic facial expressions for emotion recognition. By analyzing how random facial movements are categorized, they've identified key spatial (Action Unit) and temporal patterns that form a low-dimensional "map" of emotions, revealing how confusions arise and can be resolved.

What Problem Does This Paper Address?

Understanding how the brain processes complex, high-dimensional sensory information (like a dynamic face) into simple, meaningful categories (like emotions) is a fundamental challenge in neuroscience. Specifically, this paper tackles how we recognize the six basic facial expressions of emotion from dynamic movements. While we know certain facial muscle movements (Action Units or AUs) are involved, how these movements unfold over time and combine to form distinct emotional expressions, and how this relates to human perception and common confusions (e.g., fear vs. surprise), remains unclear. The authors aim to uncover the underlying low-dimensional representation the brain uses for this categorization.

Methods

The core of this work is the space-by-time manifold decomposition, a novel mathematical framework. It's built upon Non-negative Matrix Factorization (NMF) and Linear Discriminant Analysis (LDA).

  • NMF is used to decompose the high-dimensional facial expression data (combinations of AUs over time) into a set of non-negative spatial (AU) components and temporal components. This ensures interpretability, as face movements are additive.
  • LDA then identifies the specific combinations of these spatial and temporal components that are most effective for discriminating between the six emotion categories.
  • The authors generated stimuli using a Generative Face Grammar (GFG), which creates random dynamic facial movements by combining various AUs with random temporal parameters (onset, acceleration, etc.). This allows them to probe the perceptual space systematically.

Datasets

The study primarily uses a dataset of dynamic facial animations generated by the Generative Face Grammar (GFG). These animations were created to represent random combinations of Action Units (AUs) with varying temporal dynamics. The authors collected subjective responses from 60 Western observers who categorized these animations. A validation experiment used a smaller set of observers (10) to test the perceptual validity of the derived representations.

Key Insights and Ideas

  • Facial expressions are low-dimensional manifolds: The complex dynamics of facial movements can be effectively represented in a much lower-dimensional space, a "manifold," that captures the essential information for emotion categorization.
  • Space and time are separable but crucial: Both the specific combination of facial muscles (space) and their temporal unfolding (time) are critical for distinguishing emotions. The decomposition successfully separates these two aspects.
  • Confusions arise from shared components: Common confusions between emotions (e.g., fear and surprise) are explained by the fact that they share similar underlying AU components.
  • Temporal sequencing resolves confusions: Altering the timing or order of these shared AU components can effectively resolve confusions and lead to more accurate emotion recognition.
  • "Synergistic" AU patterns: The method identifies combinations of AUs that act together (synergies) over time, rather than just individual AUs, as key discriminators of emotion.

Key Results

  • Effective Categorization: The space-by-time manifold decomposition achieved an average categorization accuracy of 66% ± 1% across the six emotions.
  • Reduced Dimensions: The authors found that nine AU components and two temporal components were sufficient to capture the information needed to discriminate the six emotions, a significant reduction from the initial high-dimensional input.
  • Surprise vs. Fear: The study identified specific AU components (e.g., Upper Lid Raiser (AU5) and Mouth Stretch-Outer Brow Raiser (AU27-2)) and their temporal dynamics that discriminate surprise from fear. For instance, AU5 is common to both, leading to confusions, but its timing relative to other AUs can differentiate them.
  • Disgust vs. Anger: Similar findings were reported for disgust and anger, with Nose Wrinkler (AU9) being a shared component, and other AU combinations resolving the confusion.
  • Temporal Dynamics Impact: Inverted temporal dynamics (changing the sequence of AU activations) significantly affected categorization accuracy and response times for certain emotions, particularly fear, surprise, disgust, and anger. For example, inverting dynamics led to slower response times for correct rejections of happy when presented with surprise.

Limitations and Future Work

  • Observer Bias: The study used primarily Western observers, and the authors acknowledge that cultural differences might influence facial expression perception. Future work could explore cross-cultural validity.
  • Stimulus Generation: The use of random facial movements, while effective for probing the perceptual space, might not perfectly reflect the natural dynamics of spontaneous emotional expressions.
  • Generative Space for New Emotions: The authors suggest that future work could explore using the identified components as a generative space to create new emotion categories or more nuanced expressions.
  • Holistic Coding and Development: The paper raises questions about how these "unitary dimensions" of coding are learned developmentally and how attention plays a role.
  • Spatial Frequencies and Viewing Distances: While not directly tested here, the authors note that their previous work on spatial frequencies and viewing distances could be integrated with dynamic information to understand expression recognition more broadly.

Summary updated

Abstract

Visual categorization is the brain computation that reduces high-dimensional information in the visual environment into a smaller set of meaningful categories. An important problem in visual neuroscience is to identify the visual information that the brain must represent and then use to categorize visual inputs. Here we introduce a new mathematical formalism-termed space-by-time manifold decomposition-that describes this information as a low-dimensional manifold separable in space and time. We use this decomposition to characterize the representations used by observers to categorize the six classic facial expressions of emotion (happy, surprise, fear, disgust, anger, and sad). By means of a Generative Face Grammar, we presented random dynamic facial movements on each experimental trial and used subjective human perception to identify the facial movements that correlate with each emotion category. When the random movements projected onto the categorization manifold region corresponding to one of the emotion categories, observers categorized the stimulus accordingly; otherwise they selected "other." Using this information, we determined both the Action Unit and temporal components whose linear combinations lead to reliable categorization of each emotion. In a validation experiment, we confirmed the psychological validity of the resulting space-by-time manifold representation. Finally, we demonstrated the importance of temporal sequencing for accurate emotion categorization and identified the temporal dynamics of Action Unit components that cause typical confusions between specific emotions (e.g., fear and surprise) as well as those resolving these confusions.
BibTeX
@article{delis2016space,
  author = {Ioannis Delis and Chaona Chen and Rachael E. Jack and Oliver Garrod and Stefano Panzeri and Philippe G. Schyns},
  title = {{Space-by-time manifold representation of dynamic facial expressions for emotion categorization}},
  journal = {Journal of Vision},
  year = {2016},
  doi = {10.1167/16.8.14},
}
Figure