Abstract
Classifying patients' affect is a pivotal part of the mental status examination. However, this common practice is often widely inconsistent between raters. Recent advances in the field of Facial Action Recognition (FAR) have enabled the development of tools that can act to identify facial expressions from videos. In this study, we aimed to explore the potential of using machine learning techniques on FAR features extracted from videotaped semi-structured psychiatric interviews of 25 male schizophrenia inpatients (mean age 41.2 years, STD = 11.4). Five senior psychiatrists rated patients' affect based on the videos. Then, a novel computer vision algorithm and a machine learning method were used to predict affect classification based on each psychiatrist affect rating. The algorithm is shown to have a significant predictive power for each of the human raters. We also found that the eyes facial area contributed the most to the psychiatrists' evaluation of the patients' affect. This study serves as a proof-of-concept for the potential of using the machine learning FAR system as a clinician-supporting tool, in an attempt to improve the consistency and reliability of mental status examination.
Introduction
In clinical psychiatry, the Mental Status Examination (MSE) is a fundamental diagnostic tool used to assess a patient's psychological functioning. A key component of the MSE is the evaluation of affect—the externally observable, ever-changing emotional state of a patient. Unlike "mood," which describes a patient's internal, sustained feeling, affect is a dynamic expression that a clinician must observe and categorize (e.g., as "flat," "restricted," or "labile").
The primary challenge in assessing affect is its inherent subjectivity. Even senior psychiatrists often exhibit low inter-rater reliability, with significant disagreement on how to label specific affective displays. This inconsistency can cloud clinical judgment and complicate psychiatric research. This study explores whether computer vision and machine learning can provide a more objective framework by successfully "learning" the patterns that lead specific clinicians to their conclusions.
Methods
The researchers recorded semi-structured interviews with 25 male schizophrenia patients (mean age 41.2). These videos were analyzed using a multi-stage computational pipeline.
Facial Action Recognition (FAR) and Feature Engineering
The system first detected faces and aligned them based on landmarks such as the eyes, nose, and mouth. To capture the dynamic nature of affect, the team extracted "mid-level features":
- Expression Scores: Probabilities for seven emotions (anger, disgust, fear, happiness, sadness, surprise, and neutral).
- Label Variability: A measure of how frequently the dominant expression changed during the video.
- Motion Features: The standard deviation of gray-level intensities, capturing the raw variance of facial movement.
Classification and Spatial Analysis
The face was divided into six regions (eyes, cheeks, mouth, nose) to determine which areas most influenced clinical judgment. A Support Vector Machine (SVM) with a "Leave-One-Out" (LOO) cross-validation approach was used to predict the ratings of five senior psychiatrists. The FAR system was initially trained and normalized using the painDB and FEED datasets.
Results
Human Rater Variability
The study confirmed the "gap" in clinical practice: agreement between psychiatrists was moderate for affect quality (54%) but dropped significantly to 25% when categorizing specific subtypes (e.g., "suspicious" vs. "irritable").
Machine Learning Performance
Despite human inconsistency, the ML system successfully modeled individual clinical styles:
- Predictive Accuracy: The SVM achieved high accuracy in predicting individual psychiatrists' ratings, reaching up to 91.6% for "Quality of Affect" and 75% for "Range of Affect."
- Feature Significance: The eyes were identified as the most critical facial region for clinical evaluation. Furthermore, "motion" features—representing raw facial movement—were strong predictors for affect quality, suggesting that clinicians rely heavily on the overall intensity of facial activity.
Discussion
This pilot study serves as a compelling proof-of-concept that machine learning can bring a degree of standardization to subjective clinical assessments. While the system currently mimics human raters rather than establishing an absolute "truth," its high predictive power suggests that clinicians are internally consistent, even when they disagree with peers.
Limitations and Future Work
The study's primary limitations include a small, homogenous sample of male schizophrenia patients and the use of relatively short (10-minute) interview clips. Future research should include more diverse patient populations, such as those with bipolar disorder or depression, who may exhibit more salient affective displays. Ultimately, correlating FAR data with objective biological markers or long-term clinical outcomes will be necessary to develop a truly objective "gold standard" for affect classification.
Summary updated