Objective Assessment of Social Skills Using Automated Language Analysis for Identification of Schizophrenia and Bipolar Disorder

Rohit Voleti, Stephanie Woolridge, Julie M. Liss, Melissa Milanovic, Christopher R. Bowie, Visar Berisha

Introduction

Diagnosing severe mental illnesses, specifically schizophrenia and bipolar I disorder, remains a significant clinical challenge. Differential diagnosis—distinguishing between the two—is particularly difficult, with research suggesting that up to 30% of bipolar patients are initially misdiagnosed. While "thought disorders" manifested through incoherent speech are recognized hallmarks, traditional assessments rely on subjective clinical interviews. This study introduces an automated Natural Language Processing (NLP) pipeline designed to provide an objective, interpretable assessment of social functional competence, anchored to the validated Social Skills Performance Assessment (SSPA).

Methodology

The researchers developed a methodology focused on linguistic features that correlate with social skills rather than a purely data-driven "black box" approach. The pipeline involved three primary stages:

  1. Feature Extraction: 73 features were extracted across two categories:
  2. Semantic Coherence: Utilizing word and sentence embeddings (Word2Vec, Smooth Inverse Frequency, and InferSent) to measure the flow of meaning between interviewer and subject.
  3. Linguistic Complexity: Including lexical diversity (vocabulary variety), lexical density (content-to-filler word ratio), and syntactic complexity (sentence structural depth).
  4. SSPA Modeling: A greedy stepwise search identified a subset of 25 features best suited to predict manual SSPA scores assigned by human raters.
  5. Classification: Logistic Regression and Naive Bayes classifiers were trained to distinguish clinical subjects from healthy controls and to perform differential diagnosis between schizophrenia and bipolar disorder.

The study utilized transcripts from 109 subjects (43 Schizophrenia/Schizoaffective, 44 Bipolar I, and 22 Healthy Controls) performing SSPA role-playing tasks.

Results

The automated system demonstrated strong alignment with human clinical judgment. The linear regression model predicted human-assigned SSPA scores with a correlation coefficient of 0.752, as shown in Figure 1.

Linear regression of predicted vs. actual SSPA mean scores across clinical and control groups.

In classification tasks, the models achieved high accuracy. Distinguishing clinical subjects from healthy controls yielded an Area Under the Curve (AUC) of 0.96. The more challenging task of differential diagnosis between schizophrenia and bipolar disorder achieved an AUC of 0.826 using a Naive Bayes classifier.

ROC curves illustrating the classification performance for Clinical vs. Control and Schizophrenia vs. Bipolar tasks.

Key Insights

  • The "Stress Test" of Negotiation: Features from the "negotiating with a landlord" scene (Scene 3) were the most predictive. This suggests that demanding social interactions involving conflict resolution are more effective at revealing subtle language impairments.
  • Interpretability: By grounding the model in features that correlate with known clinical scores (SSPA), the researchers ensured that the AI's outputs are clinically relevant and interpretable.
  • Ensemble Power: The most effective feature sets combined simple metrics like "Bag-of-Words" averages with sophisticated deep-learning sentence encodings (InferSent), highlighting the importance of both word choice and complex structure.

Limitations and Future Work

While promising, the study is limited by a relatively small sample size (n=109). Additionally, using the same dataset for feature selection and testing may lead to slightly optimistic results. Future research will focus on refining the feature set for clinical use, subtyping individuals within diagnostic groups, and expanding the analysis to other psychiatric conditions and speech tasks.

Summary updated

Abstract

Several studies have shown that speech and language features, automatically extracted from clinical interviews or spontaneous discourse, have diagnostic value for mental disorders such as schizophrenia and bipolar disorder. They typically make use of a large feature set to train a classifier for distinguishing between two groups of interest, i.e. a clinical and control group. However, a purely data-driven approach runs the risk of overfitting to a particular data set, especially when sample sizes are limited. Here, we first down-select the set of language features to a small subset that is related to a well-validated test of functional ability, the Social Skills Performance Assessment (SSPA). This helps establish the concurrent validity of the selected features. We use only these features to train a simple classifier to distinguish between groups of interest. Linear regression reveals that a subset of language features can effectively model the SSPA, with a correlation coefficient of 0.75. Furthermore, the same feature set can be used to build a strong binary classifier to distinguish between healthy controls and a clinical group (AUC = 0.96) and also between patients within the clinical group with schizophrenia and bipolar I disorder (AUC = 0.83).
BibTeX
@inproceedings{voleti2019objective,
  author = {Rohit Voleti and Stephanie Woolridge and Julie M. Liss and Melissa Milanovic and Christopher R. Bowie and Visar Berisha},
  title = {{Objective Assessment of Social Skills Using Automated Language Analysis for Identification of Schizophrenia and Bipolar Disorder}},
  booktitle = {Interspeech},
  year = {2019},
  doi = {10.21437/interspeech.2019-2960},
}
Figure