TL;DR
This paper introduces a novel linear model, C-HiSLR, that effectively disentangles facial identity from expressions and action units using raw pixel data. By exploiting low-rank properties to implicitly remove neutral faces and enforcing group sparsity on action components, it achieves competitive performance with significantly less reliance on annotated data compared to existing methods.
What Problem Does This Paper Address?
Recognizing facial expressions and specific facial action units (AUs) is crucial for understanding human emotions and interactions. However, a major hurdle is the scarcity of large, well-annotated datasets for training complex deep learning models. Existing methods often struggle with disentangling identity, pose, and expression, and many require explicit, and often imperfect, pre-processing steps to isolate neutral faces from expressive ones. This paper aims to develop a more robust and data-efficient linear model that can learn these representations directly from raw facial video data.
Methods
The core of this work is the Collaborative-Hierarchical Sparse and Low-Rank (C-HiSLR) model. It builds upon the idea of Sparse Representation Classification (SRC) but introduces two key innovations:
- Implicit Neutral Face Subtraction: Instead of explicitly removing the neutral face, C-HiSLR models it as a low-rank component across video frames. This allows the model to learn the underlying neutral face implicitly, which is then subtracted from the observed expressive face.
- Group Sparsity for Action Components: The remaining "residual" expression components are modeled using a sparse representation. Crucially, C-HiSLR enforces group sparsity on these components. This means that related action units are encouraged to be represented together, reflecting the hierarchical nature of facial expressions.
The model is formulated as an optimization problem that is solved using the Alternating Direction Method of Multipliers (ADMM). For holistic expression recognition, the model is referred to as SLR (Sparse and Low-Rankness) when the group sparsity term is omitted.
Datasets
- Extended Cohn-Kanade (CK+) dataset: Used for evaluating holistic facial expression recognition. The authors note that they do not explicitly provide neutral faces, and the model learns from raw pixel intensities.
- MPI Face Video Database (MPI-VDB): Used for evaluating facial action unit (AU) recognition. This dataset is described as more challenging due to its profile views and longer videos.
Key Insights and Ideas
- Implicit disentanglement is powerful: The paper demonstrates that explicitly separating identity and expression is not always necessary. By modeling the neutral face as a low-rank component, the model can implicitly disentangle it from the expressive components.
- Group sparsity mirrors facial structure: Enforcing group sparsity on action units is a more natural way to represent facial expressions, as AUs often co-occur. This hierarchical structure helps improve recognition accuracy.
- Raw pixels can be sufficient: The model achieves competitive results using only raw pixel intensities, reducing the need for complex feature engineering or manual pre-processing steps like landmark detection.
- Data efficiency: The linear nature of the model makes it less demanding in terms of training data, which is a significant advantage given the scarcity of annotated facial expression datasets.
Key Results
- Holistic Expression Recognition (CK+): C-HiSLR achieved a recognition rate of 0.80, comparable to SRC (0.80) which used manually prepared expression components and neutral faces. This is a significant achievement as C-HiSLR operates on raw pixels without explicit neutral face subtraction. SLR (without group sparsity) achieved 0.70.
- Facial Action Unit Recognition (MPI-VDB): C-HiSLR achieved an average recognition rate of 0.84, outperforming SLR (0.80). This highlights the benefit of group sparsity for AU recognition.
- Comparison to SRC: The paper notes that their C-HiSLR on raw faces is comparable to SRC applied on manually prepared action components, and even better than SRC in terms of true positive rate.
Limitations and Future Work
- Temporal Modeling: The authors acknowledge that their current models do not explicitly capture the dynamics of facial expressions over time, unlike temporal or spatio-temporal models. They mention that their approach is based on appearance, and future work could explore incorporating temporal dynamics.
- Linearity Assumption: While the linear model is effective and data-efficient, it might not capture highly complex, non-linear relationships in facial expressions.
- Parameter Tuning: The performance of the model depends on tuning parameters like $\lambda_L$ and $\lambda_G$. The paper mentions that these parameters need to be tuned.
The paper suggests that the success in disentangling action components from raw pixels opens avenues for approximating expressions using piecewise linear models and encourages further exploration into more sophisticated temporal modeling.
Summary updated