Gender Fairness of Machine Learning Algorithms for Pain Detection

Dylan Green, Yuting Shang, Jiaee Cheong, Yang Liu, Hatice Gunes

Introduction

Automated pain detection systems represent a critical frontier in clinical AI, offering a voice to patients who cannot communicate, such as those in intensive care or suffering from advanced dementia. By analyzing facial expressions in real-time, these systems can alert healthcare providers to patient distress. However, the deployment of such technology is fraught with risks of automating existing societal biases. Historically, women's pain has been frequently underestimated or dismissed by clinicians. This study investigates the "research gap" in gender fairness within automated pain detection, evaluating whether high-performing algorithms maintain equitable performance across genders.

Methodology

The research compared four distinct architectural approaches to evaluate the trade-offs between raw performance and algorithmic fairness:

  1. Traditional Machine Learning: Linear Support Vector Machine (L-SVM) and Radial Basis Function SVM (RBF SVM) utilizing Histogram of Oriented Gradients (HOG) features.
  2. Deep Learning: A Convolutional Neural Network (ResNet-50) and a Vision Transformer (ViT).

To address the significant class imbalance inherent in pain datasets—where "no-pain" frames vastly outnumber "pain" frames—the researchers employed the Synthetic Minority Over-sampling Technique (SMOTE) to generate synthetic training examples.

Evaluation extended beyond standard accuracy metrics to include seven fairness indicators, most notably:

  • Equal Opportunity: Ensuring consistent true positive rates across genders.
  • Demographic Parity: Assessing if the model predicts "pain" with equal probability for both men and women.
  • Disparate Impact: Measuring the ratio of positive outcomes between groups.

Dataset and Preprocessing

The study utilized the UNBC-McMaster Shoulder Pain Expression Archive Database, comprising 48,398 frames from 25 subjects (13 female, 12 male). Pain levels were labeled using the Prkachin and Solomon Pain Intensity (PSPI) metric, which quantifies specific facial "Action Units" such as brow lowering or eye closing.

Sample frame from the UNBC-McMaster dataset showing facial features used for pain intensity labeling.

The data was partitioned using a 60/20/20 split (train/validation/test) with stratified sampling to ensure balanced participant representation across all sets.

Key Findings

The Accuracy-Fairness Paradox

The Vision Transformer (ViT) emerged as the top performer in terms of raw metrics, achieving an overall accuracy of 98.06% and a ROC AUC of 0.9969. It also demonstrated superior fairness in terms of Equal Opportunity (0.0268) and Equal Odds (0.0169) (where 0.00 represents perfect fairness).

However, the ViT did not dominate all fairness categories. The Linear SVM, despite its lower overall accuracy (95.09%), achieved the best Demographic Parity (0.1175) and Treatment Equality (0.0030). This suggests that simpler models with less complex decision boundaries may be less susceptible to learning biased correlations hidden within high-dimensional data.

Unintended Consequences of Data Balancing

A significant insight from the study was the "SMOTE side effect." While SMOTE successfully balanced the pain vs. no-pain classes, it inadvertently introduced a gender imbalance. Because the source dataset contained more female pain samples, the synthetic generation process produced a disproportionate number of female images, potentially embedding a new source of demographic bias during the training phase.

Conclusion and Future Directions

The results underscore that fairness in clinical AI is multi-dimensional; no single model excelled across all seven fairness metrics. While deep learning models generally provide higher accuracy and better error-rate parity, traditional models may offer better outcome parity.

Future research must expand to larger, more diverse datasets to ensure these findings generalize across broader populations. Furthermore, there is an urgent need to explore intersectionality—how gender bias interacts with race and age—and to implement active algorithmic mitigation strategies, such as adversarial de-biasing, to move from merely measuring bias to actively reducing it in clinical settings.

Summary updated

BibTeX
@inproceedings{green2025gender,
  author = {Dylan Green and Yuting Shang and Jiaee Cheong and Yang Liu and Hatice Gunes},
  title = {{Gender Fairness of Machine Learning Algorithms for Pain Detection}},
  booktitle = {FG},
  year = {2025},
  doi = {10.1109/fg61629.2025.11099137},
}
Figure