Predicting Affect Classification in Mental Status Examination Using Machine Learning Face Action Recognition System: A Pilot Study in Schizophrenia Patients

Ran Barzilay, Nadav Israel, Amir Krivoy, Roi Sagy, Shiri Kamhi-Nesher, Oren Loebstein, Lior Wolf, Gal ShovalFrontiers in Psychiatry 2019 · DOI

Abstract

Classifying patients' affect is a pivotal part of the mental status examination. However, this common practice is often widely inconsistent between raters. Recent advances in the field of Facial Action Recognition (FAR) have enabled the development of tools that can act to identify facial expressions from videos. In this study, we aimed to explore the potential of using machine learning techniques on FAR features extracted from videotaped semi-structured psychiatric interviews of 25 male schizophrenia inpatients (mean age 41.2 years, STD = 11.4). Five senior psychiatrists rated patients' affect based on the videos. Then, a novel computer vision algorithm and a machine learning method were used to predict affect classification based on each psychiatrist affect rating. The algorithm is shown to have a significant predictive power for each of the human raters. We also found that the eyes facial area contributed the most to the psychiatrists' evaluation of the patients' affect. This study serves as a proof-of-concept for the potential of using the machine learning FAR system as a clinician-supporting tool, in an attempt to improve the consistency and reliability of mental status examination.

Introduction

In clinical psychiatry, the Mental Status Examination (MSE) is a fundamental diagnostic tool used to assess a patient's psychological functioning. A key component of the MSE is the evaluation of affect—the externally observable, ever-changing emotional state of a patient. Unlike "mood," which describes a patient's internal, sustained feeling, affect is a dynamic expression that a clinician must observe and categorize (e.g., as "flat," "restricted," or "labile").

The primary challenge in assessing affect is its inherent subjectivity. Even senior psychiatrists often exhibit low inter-rater reliability, with significant disagreement on how to label specific affective displays. This inconsistency can cloud clinical judgment and complicate psychiatric research. This study explores whether computer vision and machine learning can provide a more objective framework by successfully "learning" the patterns that lead specific clinicians to their conclusions.

Methods

The researchers recorded semi-structured interviews with 25 male schizophrenia patients (mean age 41.2). These videos were analyzed using a multi-stage computational pipeline.

Facial Action Recognition (FAR) and Feature Engineering

The system first detected faces and aligned them based on landmarks such as the eyes, nose, and mouth. To capture the dynamic nature of affect, the team extracted "mid-level features":

  • Expression Scores: Probabilities for seven emotions (anger, disgust, fear, happiness, sadness, surprise, and neutral).
  • Label Variability: A measure of how frequently the dominant expression changed during the video.
  • Motion Features: The standard deviation of gray-level intensities, capturing the raw variance of facial movement.
Grayscale facial alignment and feature extraction used by the FAR system.

Classification and Spatial Analysis

The face was divided into six regions (eyes, cheeks, mouth, nose) to determine which areas most influenced clinical judgment. A Support Vector Machine (SVM) with a "Leave-One-Out" (LOO) cross-validation approach was used to predict the ratings of five senior psychiatrists. The FAR system was initially trained and normalized using the painDB and FEED datasets.

Temporal feature extraction showing the variance of facial signals over time.

Results

Human Rater Variability

The study confirmed the "gap" in clinical practice: agreement between psychiatrists was moderate for affect quality (54%) but dropped significantly to 25% when categorizing specific subtypes (e.g., "suspicious" vs. "irritable").

Machine Learning Performance

Despite human inconsistency, the ML system successfully modeled individual clinical styles:

  • Predictive Accuracy: The SVM achieved high accuracy in predicting individual psychiatrists' ratings, reaching up to 91.6% for "Quality of Affect" and 75% for "Range of Affect."
  • Feature Significance: The eyes were identified as the most critical facial region for clinical evaluation. Furthermore, "motion" features—representing raw facial movement—were strong predictors for affect quality, suggesting that clinicians rely heavily on the overall intensity of facial activity.

Discussion

This pilot study serves as a compelling proof-of-concept that machine learning can bring a degree of standardization to subjective clinical assessments. While the system currently mimics human raters rather than establishing an absolute "truth," its high predictive power suggests that clinicians are internally consistent, even when they disagree with peers.

Limitations and Future Work

The study's primary limitations include a small, homogenous sample of male schizophrenia patients and the use of relatively short (10-minute) interview clips. Future research should include more diverse patient populations, such as those with bipolar disorder or depression, who may exhibit more salient affective displays. Ultimately, correlating FAR data with objective biological markers or long-term clinical outcomes will be necessary to develop a truly objective "gold standard" for affect classification.

Summary updated

Abstract

Classifying patients' affect is a pivotal part of the mental status examination. However, this common practice is often widely inconsistent between raters. Recent advances in the field of Facial Action Recognition (FAR) have enabled the development of tools that can act to identify facial expressions from videos. In this study, we aimed to explore the potential of using machine learning techniques on FAR features extracted from videotaped semi-structured psychiatric interviews of 25 male schizophrenia inpatients (mean age 41.2 years, STD = 11.4). Five senior psychiatrists rated patients' affect based on the videos. Then, a novel computer vision algorithm and a machine learning method were used to predict affect classification based on each psychiatrist affect rating. The algorithm is shown to have a significant predictive power for each of the human raters. We also found that the eyes facial area contributed the most to the psychiatrists' evaluation of the patients' affect. This study serves as a proof-of-concept for the potential of using the machine learning FAR system as a clinician-supporting tool, in an attempt to improve the consistency and reliability of mental status examination.
BibTeX
@inproceedings{barzilay2019predicting,
  author = {Ran Barzilay and Nadav Israel and Amir Krivoy and Roi Sagy and Shiri Kamhi-Nesher and Oren Loebstein and Lior Wolf and Gal Shoval},
  title = {{Predicting Affect Classification in Mental Status Examination Using Machine Learning Face Action Recognition System: A Pilot Study in Schizophrenia Patients}},
  booktitle = {Frontiers in Psychiatry},
  year = {2019},
  doi = {10.3389/fpsyt.2019.00288},
}
Figure