TL;DR
This paper introduces C-HiSLR, a novel model for visual emotion recognition that leverages hierarchical sparse and low-rank representations. Unlike previous methods requiring explicit neutral faces, C-HiSLR implicitly subtracts neutral expressions and collaboratively models emotion components across multiple frames, achieving competitive performance with state-of-the-art methods.
What Problem Does This Paper Address?
Recognizing human emotions from visual data, such as videos of facial expressions, is a challenging task. Existing methods often rely on having a "neutral" face as a reference point to isolate the expression component, which can be difficult to obtain or prepare manually. Furthermore, many approaches treat each frame independently, potentially losing valuable temporal information. This paper aims to address these limitations by developing a model that can effectively recognize emotions from raw expressive facial frames without explicit neutral face preparation, while simultaneously exploiting the temporal and collaborative nature of facial expressions.
Methods
The core of the proposed method is the Collaborative-Hierarchical Sparse and Low-Rank (C-HiSLR) model. It builds upon the idea of separating an expressive face into a dominant neutral face and a sparse expression component.
- Low-Rank Property for Neutral Face Subtraction: The model assumes that a sequence of neutral faces can be represented by a low-rank matrix. By exploiting this property, C-HiSLR implicitly subtracts the neutral component from expressive frames.
- Sparse Representation for Expression Components: The remaining expression components are then modeled using a sparse representation. This means that the expression is represented as a linear combination of a few "atoms" from a learned dictionary of emotions.
- Hierarchical Sparsity: C-HiSLR introduces a hierarchical sparsity structure. It enforces group sparsity across different frames or spatial locations that are related to the same emotion category, and atom-wise sparsity within each frame to capture fine-grained expression details. This hierarchical approach is motivated by the observation that signals over frames share class-wise, but not necessarily atom-wise, sparsity patterns.
- Collaborative Modeling: The "Collaborative" aspect refers to jointly learning the sparse and low-rank components across multiple channels (frames) to capture inter-channel dependencies.
-
SLR (Sparse and Low-Rank Representation): As a precursor to C-HiSLR, the paper introduces the SLR model, which jointly optimizes for sparse representation (X) and a low-rank term (L) representing the neutral face, formulated as: $$ \min_{X,L} \|X\|_1 + \lambda_L \|L\|_* \quad \text{s.t.} \quad Y = DX + L $$ where $Y$ is the input expressive face, $D$ is the dictionary of emotions, $\|X\|_1$ is the L1 norm promoting sparsity in $X$, and $\|L\|_*$ is the nuclear norm promoting low-rankness in $L$.
-
C-HiSLR Extension: C-HiSLR extends SLR by incorporating group sparsity on the representation matrix $X$: $$ \min_{X,L} \|X\|_1 + \lambda_L \|L\|_* + \lambda_G \sum_{G \in \mathcal{G}} \|X[G]\|_F \quad \text{s.t.} \quad Y = DX + L $$ Here, $\mathcal{G}$ represents a partition of the dictionary indices into groups, and $\|X[G]\|_F$ is the Frobenius norm of the sub-matrix corresponding to group $G$, enforcing group sparsity.
-
Optimization: The models are optimized using an iterative Alternating Direction Method of Multipliers (ADMM) approach.
Datasets
- CK+ Dataset: All experiments were conducted on the Extended Cohn-Kanade (CK+) dataset, which contains 321 emotion sequences labeled with seven basic emotions (angry, contempt, disgust, fear, happiness, sadness, surprise). The dataset was randomly split into training (10 sequences per category) and testing (5 sequences per category) sets.
Key Insights and Ideas
- Implicit Neutral Face Subtraction: The most significant insight is the ability to implicitly handle neutral faces by exploiting their low-rank property, eliminating the need for explicit preparation or knowledge of neutral frames.
- Hierarchical Sparsity for Robustness: The hierarchical sparsity (group sparsity + atom-wise sparsity) is crucial. Group sparsity helps to enforce consistency across related frames or spatial regions, leading to more robust emotion recognition, especially when dealing with imperfect alignment.
- Collaborative Modeling Captures Temporal Dynamics: By treating video frames as a multichannel signal and modeling them collaboratively, the method effectively captures the temporal dynamics of facial expressions, which are essential for accurate emotion recognition.
- Emotion as a Sparse Representation: The core idea that emotions can be sparsely represented by a dictionary of emotion "atoms" is validated, offering an alternative to traditional subspace methods.
Key Results
- Competitive Performance: C-HiSLR achieves a total recognition rate of 0.80 on the CK+ dataset, which is comparable to the Sparse Representation based Classification (SRC) method that explicitly uses neutral faces.
- Improved Sensitivity: C-HiSLR demonstrates superior performance in terms of sensitivity (true positive rate) compared to SRC, indicating its effectiveness in correctly identifying positive instances of each emotion.
- Effectiveness of Group Sparsity: The paper highlights that C-HiSLR's group sparsity regularization significantly boosts performance, particularly in sensitivity, validating the importance of this hierarchical structure.
- Robustness to Alignment: Qualitative results (Fig. 5) suggest that C-HiSLR is more robust to imperfect facial alignment than methods that do not enforce group sparsity.
Limitations and Future Work
- Imperfect Alignment: The paper acknowledges that imperfect alignment can still affect performance, as shown in Fig.
- Future work is suggested to address misalignment more explicitly.
- Dictionary Learning: The current model uses a fixed dictionary. The authors suggest incorporating dictionary learning as a future direction to adapt the dictionary to specific datasets or conditions.
- Handling Misalignment: Explicitly handling misalignment is identified as a key area for future research.
Summary updated