TL;DR
This paper introduces WSDA-OR, a novel deep learning framework for estimating pain intensity from facial expressions in videos. It tackles the challenge of domain shift and limited annotations by combining weakly-supervised learning, domain adaptation, and ordinal regression, outperforming existing methods in accurately localizing pain.
What Problem Does This Paper Address?
Estimating pain intensity from facial expressions holds significant promise for healthcare, especially for individuals unable to communicate verbally. However, a major hurdle is the "domain shift" – the difference in data characteristics between controlled lab settings (source domain) and real-world scenarios (target domain). Collecting and meticulously labeling pain intensity data for every frame is extremely laborious and prone to subjective bias. To address this, researchers have turned to weakly-supervised learning (WSL), where only coarse labels (e.g., sequence-level) are available. While WSL has shown promise, existing methods often treat pain intensity as a continuous regression problem or fail to fully leverage the inherent ordinal relationship between pain levels (e.g., level 3 is more painful than level 2, but less than level 4). Furthermore, many WSL approaches only consider a single "peak" frame within a video sequence, potentially missing crucial information from adjacent frames. This paper aims to bridge these gaps by developing a method that effectively handles domain shift with weakly-labeled data, explicitly models the ordinal nature of pain, and utilizes information from multiple relevant frames within a sequence.
Methods
The core of the proposed WSDA-OR model lies in its integration of several key techniques:
- Weakly-Supervised Domain Adaptation (WSDA): The model leverages a fully-labeled source domain (RECOLA) to train a feature extractor and then adapts it to a weakly-labeled target domain (UNBC-McMaster, BIOVID, Fatigue). This adaptation is achieved through an adversarial mechanism where a domain discriminator tries to distinguish between source and target features, while the feature extractor is trained to fool the discriminator, thus learning domain-invariant representations.
- Ordinal Regression with Gaussian Modeling: Instead of treating pain intensity as discrete categories or continuous values, WSDA-OR models the ordinal relationship. Crucially, it uses "soft Gaussian labels" to represent these ordinal levels. This approach is more robust than hard assignments and allows the model to capture the relative ordering of pain intensities more effectively. The mean of the Gaussian represents the label, and the variance accounts for the influence of neighboring intensity levels.
- Adaptive Multiple Instance Learning (MIL) Pooling: Traditional MIL approaches often use a "max pooling" strategy, selecting only the frame with the highest predicted intensity within a sequence. WSDA-OR employs an "adaptive MIL pooling" mechanism that considers multiple relevant frames from a sequence to infer the sequence-level label. This is achieved by identifying frames predicted to have high intensity and aggregating their features, thereby utilizing more comprehensive temporal information.
- Deep Architecture: The model utilizes a deep neural network architecture, specifically an I3D (Inflated 3D ConvNet) model, which is adept at capturing spatio-temporal features in videos. This architecture is composed of a feature mapping network, a label predictor, and a domain classifier.
Datasets
The proposed WSDA-OR model was evaluated on several datasets:
- RECOLA: Used as the fully-labeled source domain. This dataset contains video sequences with frame-level annotations of pain intensity.
- UNBC-McMaster: Used as a weakly-labeled target domain. This dataset is commonly used for pain intensity estimation and contains videos with sequence-level labels.
- BIOVID and Fatigue (private): Also used as weakly-labeled target domains for sequence-level estimation.
Key Insights and Ideas
- Ordinality Matters: Explicitly modeling the ordinal relationship between pain intensity levels, rather than treating them as independent categories or continuous values, significantly improves estimation accuracy. The Gaussian representation is a clever way to encode this ordinality in a soft, learnable manner.
- Beyond Peak Frames: Relying on a single "peak" frame for sequence-level prediction is suboptimal. Utilizing multiple relevant frames through adaptive MIL pooling allows the model to capture richer temporal dynamics and context, leading to better performance.
- Adversarial DA for Weak Labels: Combining adversarial domain adaptation with weakly-supervised learning is a powerful strategy for bridging the gap between controlled and real-world data, especially when fine-grained labels are scarce in the target domain.
- Improved Pain Localization: The WSDA-OR approach demonstrates superior ability to localize pain within video sequences compared to previous methods, even outperforming ground truth in some instances where ground truth might be less precise.
Key Results
The paper reports significant improvements over state-of-the-art methods:
- UNBC-McMaster Dataset: The proposed WSDA-OR achieved a PCC (Pearson Correlation Coefficient) of 0.705 and an ICC (Intraclass Correlation Coefficient) of 0.696 for frame-level estimation, and a PCC of 0.745 and ICC of 0.750 for sequence-level estimation. These metrics represent a substantial improvement over existing weakly-supervised domain adaptation methods like WSDA [12] and other MIL approaches.
- BIOVID and Fatigue Datasets: WSDA-OR also demonstrated superior performance on these datasets compared to baseline transfer learning approaches, highlighting its effectiveness in generalizing to different weakly-labeled target domains.
- Qualitative Results: Visualizations in Figure 2 show that WSDA-OR can accurately localize pain even in scenarios where ground truth might be less clear or where there are subtle pain expressions.
Limitations and Future Work
- Computational Cost: While not explicitly detailed as a limitation, deep learning models, especially those involving adversarial training and 3D CNNs, are computationally intensive.
- Subjectivity of Pain: Pain is inherently subjective, and while the model aims to capture objective facial cues, the interpretation of pain intensity can still be influenced by individual differences.
- Future Work: The authors suggest further exploration of the proposed framework for other applications beyond pain estimation. The paper also implies that further refinement of the MIL pooling mechanism and Gaussian modeling could yield even better results. The potential for exploring more complex ordinal regression techniques within the WSDA framework is also an open avenue.
Summary updated