Abstract
In this paper, we design a Collaborative-Hierarchical Sparse and Low-Rank (C-HiSLR) model that is natural for recognizing human emotion in visual data. Previous attempts require explicit expression components, which are often unavailable and difficult to recover. Instead, our model exploits the low-rank property to subtract neutral faces from expressive facial frames as well as performs sparse representation on the expression components with group sparsity enforced. For the CK+ dataset, C-HiSLR on raw expressive faces performs as competitive as the Sparse Representation based Classification (SRC) applied on manually prepared emotions. Our C-HiSLR performs even better than SRC in terms of true positive rate.
Introduction
In traditional automated emotion recognition, the standard approach involves subtracting a "neutral" face from an "expressive" face to isolate the specific muscle movements associated with an emotion, such as a smile or a frown. However, in real-world scenarios, a clean neutral frame is not always available or easily identifiable. This creates a significant bottleneck for automated systems that must process raw video data.
Researchers at Johns Hopkins University have addressed this challenge by developing the Collaborative-Hierarchical Sparse and Low-Rank (C-HiSLR) model. This approach moves away from manual preparation, focusing instead on automatic separation: distinguishing between a person's permanent identity (the static component) and their fleeting expression (the dynamic component) directly from raw video frames.
Methodology
The core of the C-HiSLR model is a mathematical decomposition of the video matrix $Y$. The model treats a video sequence as a multichannel signal and decomposes it into two primary components:
- Low-Rank Representation ($L$): The model assumes that the neutral face—representing the subject's identity—remains largely invariant across all frames of a video. In matrix terms, this component is "low-rank."
- Sparse Representation ($DX$): The emotion is treated as a sparse signal, representing localized changes relative to the neutral face. This is modeled using a dictionary ($D$) of known emotions and a sparse coefficient matrix ($X$).
To ensure robustness, the model enforces group sparsity (the "Collaborative" aspect). Since a video clip typically contains multiple frames of the same emotion, this constraint ensures that all frames in a sequence are collaboratively classified into the same category, preventing the model from flickering between different emotions due to noise or blur. The optimization is solved using the Alternating Direction Method of Multipliers (ADMM), which efficiently extracts both components simultaneously.
Experimental Setup
The model was evaluated using the Extended Cohn-Kanade (CK+) dataset, a standard benchmark containing 321 sequences across seven emotion categories: Anger (An), Contempt (Co), Disgust (Di), Fear (Fe), Happiness (Ha), Sadness (Sa), and Surprise (Su).
The study compared the "blind" C-HiSLR approach (which receives no neutral face information) against standard Sparse Representation-based Classification (SRC). Notably, the SRC baseline was given an advantage: it was tested on manually prepared data where the neutral face had already been subtracted.
Key Results and Insights
Despite the lack of manual pre-processing, C-HiSLR achieved a total recognition rate of 0.80, matching the performance of the manually-assisted SRC. More importantly, C-HiSLR demonstrated a significant boost in sensitivity (true positive rate) for several difficult emotion categories.
As shown in the results table, C-HiSLR significantly outperformed SRC in identifying "Contempt" (0.84 vs 0.60), "Fear" (0.53 vs 0.25), and "Sadness" (0.65 vs 0.24). These results suggest that by integrating the low-rank subtraction directly into the classification objective, the model reduces error propagation and more accurately isolates the subtle signals of these complex emotions.
Conclusion and Future Work
The C-HiSLR model proves that emotion is automatically separable from expressive faces using raw pixel intensities, eliminating the need for complex feature engineering or manual neutral-frame selection.
Future research will focus on improving the model's resilience to alignment sensitivity, as imperfect face tracking can currently degrade performance. Additionally, the authors suggest that incorporating dictionary learning—allowing the model to adaptively learn emotion templates during training—could further enhance accuracy and generalizability across diverse datasets.
Summary updated