TL;DR
This paper introduces a novel algorithm, Quantified Facial Expressiveness (QFE), to objectively measure the intensity and dynamism of facial expressions in videos. By combining spatial (Action Unit intensities) and temporal (landmarks, head pose, gaze) features, QFE provides a bounded score that allows for more robust comparisons across datasets and individuals, addressing the limitations of manual annotation and previous unbounded methods.
What Problem Does This Paper Address?
Analyzing human emotions and affective states is crucial for understanding human behavior, but it's complicated by differing theories on whether emotions are universal or constructed from context. Facial expressions are a primary channel for conveying affect, yet quantifying their intensity and dynamism objectively and at scale remains a challenge. Current methods often rely on subjective human annotations, which are time-consuming and prone to bias, or use unbounded metrics that hinder cross-dataset comparisons. This paper aims to bridge this gap by developing an automated, objective, and bounded method for measuring facial expressiveness, enabling more scalable and reliable affective behavior analytics.
Methods
The core of this work is the Quantified Facial Expressiveness (QFE) algorithm, which integrates both spatial and temporal aspects of facial expressions.
- Spatial Expressiveness ($\sigma$): This component quantifies expressiveness within a single frame. It leverages the intensities of facial Action Units (AUs), giving more weight to AUs with higher intensities and larger temporal changes. The formula used is designed to produce a bounded score between 0 and 1.
- Temporal Expressiveness ($\delta$): To capture the dynamic nature of expressions, the algorithm uses other facial modalities like 2D/3D landmarks, head pose orientation and rotation, and gaze location. It measures the relative change (velocity) of these features between frames and approximates temporal expressiveness using a Taylor series expansion, which is then scaled to a [0, 1] range.
- Combining Spatial and Temporal Expressiveness ($\tau$): The paper proposes three approaches to combine $\sigma$ and $\delta$:
- Weighted Combination (Approach 1): Treats spatial expressiveness ($\sigma$) as the primary source and temporal expressiveness ($\delta$) as auxiliary, combining them with predefined weights.
- Weighted Combination with Adjustment (Approach 2): Allows for flexible weighting of spatial and temporal components, enabling scenarios where temporal features (like gaze) might be more critical than spatial ones.
- Latent Variable Model (Approach 3): Uses a generative model to estimate the latent facial expressiveness variable from all modalities.
The authors ultimately focus on Approach 1 for downstream tasks due to its simplicity and effectiveness in capturing overall expressiveness.
Datasets
The proposed QFE algorithm was evaluated on two publicly available datasets:
- DISFA [39]: A spontaneous facial expression dataset with 27 subjects, featuring frontal face images and AU annotations. It includes videos with varied stimuli designed to elicit expressions.
- BioVid Pain Dataset [49]: This dataset contains 90 subjects performing pain and other expressions, with continuous heat applied as a stimulus. The paper utilized part C of the dataset, which includes raw data and pain labels.
Key Insights and Ideas
- Context Matters: The research strongly suggests that external stimuli (context) significantly influence facial expressiveness, supporting the theory of constructed emotion.
- Subjectivity is Quantifiable: Even with the same context, individuals exhibit distinct patterns and intensities of facial expressiveness, a phenomenon the QFE algorithm can effectively measure.
- Multimodal Fusion is Key: Combining spatial (AU intensities) and temporal (landmarks, pose, gaze) features provides a more comprehensive and accurate measure of facial expressiveness than relying on a single modality.
- Bounded Scores Enable Comparison: The development of a bounded expressiveness score (0-1) is crucial for objectively comparing expressiveness across different individuals, datasets, and experimental conditions.
- Action Units Alone Are Insufficient: While AUs are important, temporal dynamics captured by other features are essential for a complete picture of expressiveness, especially for dynamic changes.
Key Results
- The QFE algorithm successfully captured facial expressiveness, with results showing good agreement with human annotators (average assessment between 'very good' and 'excellent').
- Granger causality tests indicated that temperature (context) could Granger-cause ground-truth pain levels in a significant percentage of video segments (up to 100%), but its causal link to facial pain expressiveness was weaker (up to 23%), suggesting that pain expression might not always directly correlate with the stimulus.
- Analysis of the DISFA dataset revealed significant subjective differences in expressiveness among individuals, even when exposed to the same stimuli. Metrics like MARPE and DTW distance showed high variability, indicating diverse response patterns.
- The study demonstrated that the QFE algorithm can capture temporal changes and subjective differences in expressiveness, providing useful insights for affective behavior analytics.
Limitations and Future Work
- Limitations:
- The paper acknowledges that the influence of noise and anomalies in automated feature extraction can negatively impact QFE scores.
- While the BioVid pain dataset showed a relationship between temperature and pain level, the direct correlation with facial pain expressiveness was not as strong, potentially due to factors like inappropriate affect or subjects not displaying their true pain.
- A quantitative comparison with prior works is difficult due to the lack of publicly available frame-level annotated datasets for expressiveness.
- Future Work:
- Dimensional Models: Incorporating dimensional models of emotion (e.g., valence, arousal) could provide a richer understanding of expressiveness.
- Mitigating Noise: Investigating techniques to mitigate noise and anomalies from feature extraction, such as leveraging confidence scores from tools like OpenFace or using anomaly detection methods, is suggested.
- Exploring Other Modalities: Further exploration of other modalities and their impact on expressiveness is encouraged.
- Automated Tool Impact: Investigating how different automated feature extraction tools affect QFE results is a potential area for future research.
Summary updated