TL;DR
This paper introduces DeepFaceLIFT, a novel two-stage personalized model for automatically estimating self-reported pain (Visual Analog Scale - VAS) from facial expressions. By moving beyond generic pain metrics and incorporating individual characteristics, DeepFaceLIFT significantly outperforms traditional methods and offers interpretable insights into how facial cues relate to pain perception.
What Problem Does This Paper Address?
Estimating pain from facial expressions is challenging because pain is inherently subjective and individuals express it differently. While previous work often focused on objective metrics like the Prkachin and Solomon Pain Intensity (PSPI) score, these may not accurately reflect an individual's true pain experience. The gold standard for pain measurement is the self-reported Visual Analog Scale (VAS), but its subjective nature makes automatic estimation difficult. Existing methods struggle to account for individual variations in facial expression, leading to a "one-size-fits-all" approach that is less effective. This paper aims to bridge this gap by developing a personalized model that directly estimates VAS scores, considering individual differences and providing interpretable results.
Methods
DeepFaceLIFT employs a two-stage hierarchical learning approach:
- Stage 1: Weakly Supervised Neural Network (NN): This stage uses a fully-connected NN to generate frame-level VAS estimates. It leverages multi-instance and multi-task learning, taking Active Appearance Model (AAM) facial landmarks as input. Crucially, this stage allows for personalization by incorporating personal features (age, gender, complexion) either by appending them to the NN's input or an intermediate layer. The model is trained to predict both VAS and, optionally, Observed Pain Intensity (OPI) scores.
- Stage 2: Gaussian Process (GP) Regression with RBF-ARD Kernel: The frame-level VAS estimates from Stage 1 are aggregated into sequence-level statistics (mean, median, variance, etc.). These statistics are then fed into a GP regression model. The RBF-ARD kernel is used to automatically determine the relevance of each feature statistic, providing a robust way to model sequence-level dependencies and estimate the final VAS score. This stage also provides uncertainty estimates for the predictions.
A key innovation is the use of DeepLIFT (a method for explaining NN predictions) in Stage 1 to understand how individual facial landmarks contribute to the pain estimation, offering interpretability.
Datasets
The primary dataset used is the UNBC-McMaster Shoulder Pain Expression Archive Database (UNBC-PAIN). This dataset contains facial videos of subjects experiencing shoulder pain during range-of-motion exercises. Each frame is annotated with PSPI scores, and each sequence is annotated with self-reported VAS scores. The paper notes that while the dataset has a large number of frames, the number of sequence-level VAS labels is comparatively limited (200), posing a challenge for estimation.
Key Insights and Ideas
- Personalization is Key: Directly estimating individual VAS scores, rather than generic metrics, and incorporating personal features significantly improves pain estimation accuracy.
- Facial Landmarks as Rich Features: AAM facial landmarks, despite reducing dimensionality from raw images, provide sufficient and intuitive information for pain estimation.
- Interpretability Matters: The DeepFaceLIFT framework, through DeepLIFT and GP kernel analysis, allows researchers to understand which facial landmarks and which feature statistics are most important for pain estimation, both generally and for individual subjects.
- OPI as a Grounding Signal: Including OPI (observer-rated pain) as an additional task in the NN training helps to ground the subjective VAS estimates, leading to improved performance.
- Feature Placement for Personalization: Appending personal features to an intermediate layer of the NN in Stage 1 yielded better results than appending them directly to the input features.
Key Results
- DeepFaceLIFT achieved a significantly lower Mean Absolute Error (MAE) for VAS estimation compared to baseline models (NN-MV, RNN, HCRF). For example, using only VAS estimates, DeepFaceLIFT achieved an MAE of 2.18, outperforming the NN-MV baseline (2.48).
- The Intra-Class Correlation (ICC) for DeepFaceLIFT improved from a baseline of 0.19 to 0.35, indicating better agreement between estimated and true VAS scores.
- The model automatically discovers pain-relevant facial regions for each person, aiding in the interpretation of pain-related facial cues.
- Analysis of feature statistics revealed that the sum, mean, and IQR of frame-level estimates were the most important statistics for the GP model.
- Personal features (age, gender, complexion) were found to improve VAS estimates, with age showing the most significant impact.
Limitations and Future Work
- Limited VAS Labels: The UNBC-McMaster dataset has a limited number of sequence-level VAS labels, which poses a challenge for training.
- Generalization to Unseen Subjects: While the model is personalized, the paper notes that some methods (like Martinez et al., 2017) may require a certain number of ratings from a new subject before generalizing well. DeepFaceLIFT aims to circumvent this by generalizing in a personalized manner without explicit prior data.
- Contextual Information: The paper acknowledges that context is an important aspect of pain perception that was not explored.
- Future Work: The authors plan to investigate the relationships between VAS and other pain scores (like PSPI), explore how these relate to facial Action Units (AUs), and derive more advanced statistics to capture additional information for improved pain estimation. They also aim to further investigate the influence of context.
Summary updated