DeepFaceLIFT: Interpretable Personalized Models for Automatic Estimation of Self-Reported Pain

Dianbo Liu, Fengjiao Peng, Andrew Shea, Ognjen Rudovic, Rosalind W. Picard

TL;DR

This paper introduces DeepFaceLIFT, a novel two-stage model that personalizes facial expression analysis to estimate an individual's self-reported pain (VAS). By combining neural networks and Gaussian processes with personal features, DeepFaceLIFT significantly outperforms generic models, offering more accurate and interpretable pain assessments.

What Problem Does This Paper Address?

Estimating pain from facial expressions is challenging because pain is a subjective experience. Traditional methods often rely on "one-size-fits-all" objective metrics (like PSPI), which don't fully capture individual pain levels. Self-reported pain, specifically the Visual Analog Scale (VAS), is considered the gold standard but is highly subjective and varies greatly between individuals due to factors like age, gender, and personal experience. Existing automated methods struggle to account for this individual variability, leading to less accurate and less trustworthy pain estimations. The paper aims to bridge this gap by developing a model that can personalize pain estimation to each individual.

Methods

DeepFaceLIFT employs a two-stage approach:

  1. Stage 1: Weakly Supervised Neural Network (NN): This stage uses a fully-connected NN to process Active Appearance Model (AAM) facial landmarks from each video frame. Crucially, it uses multi-task learning, training the NN to predict both the sequence-level VAS and, where available, the Observed Pain Intensity (OPI) score. This stage also incorporates personal features (age, gender, complexion) either by appending them to the NN's input or an intermediate layer, allowing for feature-level personalization. The DeepLIFT technique is used here to interpret how individual facial landmarks contribute to the output.
  2. Stage 2: Gaussian Process (GP) Regression: The outputs from the NN (frame-level VAS estimates) are aggregated into a set of sequence-level statistics (mean, variance, etc.). These statistics are then fed into a Gaussian Process regression model. The GP model, using an RBF-ARD kernel, learns to predict the final sequence-level VAS score and also provides uncertainty estimates for its predictions. This stage further refines the personalization by considering the aggregated facial expression information.

Datasets

The primary dataset used is the UNBC-McMaster Shoulder Pain Expression Archive Database (UNBC-PAIN). This dataset contains facial videos of individuals experiencing one-sided shoulder pain during range-of-motion exercises. It includes:

  • 200 image sequences from 25 subjects.
  • 48,398 image frames.
  • PSPI scores (based on facial action unit intensities) for each frame.
  • Observed Pain Intensity (OPI) scores (0-5 Likert-like scale) from experienced observers for each sequence.
  • Self-reported Visual Analog Scale (VAS) ratings (0-10 ordinal scale) for each sequence.

The paper notes that while the dataset is rich in frames, the number of sequence-level VAS labels is comparatively limited (200), posing a challenge for estimation.

Key Insights and Ideas

  • Personalization is Key: The core insight is that generic pain estimation models are insufficient. Personalizing the model using individual characteristics (age, gender, complexion) and their specific facial expression patterns significantly improves accuracy.
  • VAS as the Gold Standard: The paper emphasizes the importance of directly estimating self-reported VAS, which is the clinical gold standard, rather than relying solely on objective metrics like PSPI.
  • Multi-Task Learning for Robustness: Using OPI scores alongside VAS in a multi-task learning setup for the NN not only improves VAS estimation but also suggests OPI provides a valuable "grounding" signal, potentially mitigating the subjectivity of self-reports.
  • Interpretability Matters: The use of DeepLIFT allows for understanding which facial landmarks are most important for pain estimation, and how these contributions vary across individuals and between VAS and OPI estimation. This transparency is crucial for building trust and understanding in AI-driven medical assessments.
  • Feature Statistics are Informative: Aggregating frame-level predictions into statistics (like sum, mean, and IQR) for the GP model is a powerful way to capture temporal dynamics and sequence-level information, leading to better overall estimates.

Key Results

  • Performance Improvement: DeepFaceLIFT achieved a significantly lower Mean Absolute Error (MAE) for VAS estimation compared to baseline models (NN-MV, RNN, HCRF). The paper reports an MAE of 2.18 for DeepFaceLIFT, outperforming the NN-MV baseline (2.48).
  • Intra-Class Correlation (ICC): The ICC, a measure of agreement between estimated and true VAS, improved from a baseline of 19% to 35% with DeepFaceLIFT. This indicates a much stronger correlation and agreement, suggesting the model's estimates are more reliable.
  • Personal Features Impact: Including personal features improved performance across models, with age showing the most significant benefit for VAS estimation. Appending personal features to an intermediate NN layer yielded better results than appending them to the input layer.
  • Interpretability Findings: DeepLIFT revealed that different facial landmarks contribute most to VAS and OPI estimation, and these contributions vary significantly among subjects. This highlights the personalized nature of facial pain expression.

Limitations and Future Work

  • Limited VAS Labels: The paper acknowledges the challenge posed by the relatively small number of sequence-level VAS labels (200) in the UNBC-McMaster dataset.
  • Contextual Information: The authors note that context (e.g., the situation causing pain) is an important aspect not explored in this paper.
  • Future Work:
    • Investigate relationships between VAS and other pain scores (like PSPI) and facial Action Units (AUs).
  • Derive more advanced statistics to capture additional information and improve estimates.
  • Explore the role of context in pain estimation.
  • Further investigate the differences in landmark contributions and whether they reflect model behavior or inherent subject differences.

Summary updated

Abstract

Previous research on automatic pain estimation from facial expressions has focused primarily on "one-size-fits-all" metrics (such as PSPI). In this work, we focus on directly estimating each individual's self-reported visual-analog scale (VAS) pain metric, as this is considered the gold standard for pain measurement. The VAS pain score is highly subjective and context-dependent, and its range can vary significantly among different persons. To tackle these issues, we propose a novel two-stage personalized model, named DeepFaceLIFT, for automatic estimation of VAS. This model is based on (1) Neural Network and (2) Gaussian process regression models, and is used to personalize the estimation of self-reported pain via a set of hand-crafted personal features and multi-task learning. We show on the benchmark dataset for pain analysis (The UNBC-McMaster Shoulder Pain Expression Archive) that the proposed personalized model largely outperforms the traditional, unpersonalized models: the intra-class correlation improves from a baseline performance of 19\% to a personalized performance of 35\% while also providing confidence in the model\textquotesingle s estimates -- in contrast to existing models for the target task. Additionally, DeepFaceLIFT automatically discovers the pain-relevant facial regions for each person, allowing for an easy interpretation of the pain-related facial cues.
BibTeX
@inproceedings{liu2017deepfacelift,
  author = {Dianbo Liu and Fengjiao Peng and Andrew Shea and Ognjen Rudovic and Rosalind W. Picard},
  title = {{DeepFaceLIFT: Interpretable Personalized Models for Automatic Estimation of Self-Reported Pain}},
  booktitle = {AffComp @ IJCAI},
  year = {2017},
  url = {https://openreview.net/forum?id=rkbxSrG_Wr},
}

Loading related content...

Figure