TL;DR
This paper introduces DeepFaceLIFT, a novel two-stage model that personalizes facial expression analysis to estimate an individual's self-reported pain (VAS). By combining neural networks and Gaussian processes with personal features, DeepFaceLIFT significantly outperforms generic models, offering more accurate and interpretable pain assessments.
What Problem Does This Paper Address?
Estimating pain from facial expressions is challenging because pain is a subjective experience. Traditional methods often rely on "one-size-fits-all" objective metrics (like PSPI), which don't fully capture individual pain levels. Self-reported pain, specifically the Visual Analog Scale (VAS), is considered the gold standard but is highly subjective and varies greatly between individuals due to factors like age, gender, and personal experience. Existing automated methods struggle to account for this individual variability, leading to less accurate and less trustworthy pain estimations. The paper aims to bridge this gap by developing a model that can personalize pain estimation to each individual.
Methods
DeepFaceLIFT employs a two-stage approach:
- Stage 1: Weakly Supervised Neural Network (NN): This stage uses a fully-connected NN to process Active Appearance Model (AAM) facial landmarks from each video frame. Crucially, it uses multi-task learning, training the NN to predict both the sequence-level VAS and, where available, the Observed Pain Intensity (OPI) score. This stage also incorporates personal features (age, gender, complexion) either by appending them to the NN's input or an intermediate layer, allowing for feature-level personalization. The DeepLIFT technique is used here to interpret how individual facial landmarks contribute to the output.
- Stage 2: Gaussian Process (GP) Regression: The outputs from the NN (frame-level VAS estimates) are aggregated into a set of sequence-level statistics (mean, variance, etc.). These statistics are then fed into a Gaussian Process regression model. The GP model, using an RBF-ARD kernel, learns to predict the final sequence-level VAS score and also provides uncertainty estimates for its predictions. This stage further refines the personalization by considering the aggregated facial expression information.
Datasets
The primary dataset used is the UNBC-McMaster Shoulder Pain Expression Archive Database (UNBC-PAIN). This dataset contains facial videos of individuals experiencing one-sided shoulder pain during range-of-motion exercises. It includes:
- 200 image sequences from 25 subjects.
- 48,398 image frames.
- PSPI scores (based on facial action unit intensities) for each frame.
- Observed Pain Intensity (OPI) scores (0-5 Likert-like scale) from experienced observers for each sequence.
- Self-reported Visual Analog Scale (VAS) ratings (0-10 ordinal scale) for each sequence.
The paper notes that while the dataset is rich in frames, the number of sequence-level VAS labels is comparatively limited (200), posing a challenge for estimation.
Key Insights and Ideas
- Personalization is Key: The core insight is that generic pain estimation models are insufficient. Personalizing the model using individual characteristics (age, gender, complexion) and their specific facial expression patterns significantly improves accuracy.
- VAS as the Gold Standard: The paper emphasizes the importance of directly estimating self-reported VAS, which is the clinical gold standard, rather than relying solely on objective metrics like PSPI.
- Multi-Task Learning for Robustness: Using OPI scores alongside VAS in a multi-task learning setup for the NN not only improves VAS estimation but also suggests OPI provides a valuable "grounding" signal, potentially mitigating the subjectivity of self-reports.
- Interpretability Matters: The use of DeepLIFT allows for understanding which facial landmarks are most important for pain estimation, and how these contributions vary across individuals and between VAS and OPI estimation. This transparency is crucial for building trust and understanding in AI-driven medical assessments.
- Feature Statistics are Informative: Aggregating frame-level predictions into statistics (like sum, mean, and IQR) for the GP model is a powerful way to capture temporal dynamics and sequence-level information, leading to better overall estimates.
Key Results
- Performance Improvement: DeepFaceLIFT achieved a significantly lower Mean Absolute Error (MAE) for VAS estimation compared to baseline models (NN-MV, RNN, HCRF). The paper reports an MAE of 2.18 for DeepFaceLIFT, outperforming the NN-MV baseline (2.48).
- Intra-Class Correlation (ICC): The ICC, a measure of agreement between estimated and true VAS, improved from a baseline of 19% to 35% with DeepFaceLIFT. This indicates a much stronger correlation and agreement, suggesting the model's estimates are more reliable.
- Personal Features Impact: Including personal features improved performance across models, with age showing the most significant benefit for VAS estimation. Appending personal features to an intermediate NN layer yielded better results than appending them to the input layer.
- Interpretability Findings: DeepLIFT revealed that different facial landmarks contribute most to VAS and OPI estimation, and these contributions vary significantly among subjects. This highlights the personalized nature of facial pain expression.
Limitations and Future Work
- Limited VAS Labels: The paper acknowledges the challenge posed by the relatively small number of sequence-level VAS labels (200) in the UNBC-McMaster dataset.
- Contextual Information: The authors note that context (e.g., the situation causing pain) is an important aspect not explored in this paper.
- Future Work:
- Investigate relationships between VAS and other pain scores (like PSPI) and facial Action Units (AUs).
- Derive more advanced statistics to capture additional information and improve estimates.
- Explore the role of context in pain estimation.
- Further investigate the differences in landmark contributions and whether they reflect model behavior or inherent subject differences.
Summary updated