Masking Morphosyntactic Categories to Evaluate Salience for Schizophrenia Diagnosis

Yaara Shriki, Ido Ziv, Nachum Dershowitz, Eiran Harel, Kfir BarNAACL 2022

TL;DR

This paper explores how masking specific word categories (like nouns or verbs) in transcribed speech affects a machine learning model's ability to distinguish between individuals with schizophrenia and healthy controls. They found that nouns are the most critical category for accurate classification, suggesting that how individuals with schizophrenia use nouns, rather than just their frequency, is a key diagnostic indicator.

What Problem Does This Paper Address?

Schizophrenia is a complex mental disorder characterized by thought disorders, which often manifest as disorganized speech. Detecting these speech patterns computationally could aid in diagnosis, but understanding which linguistic features are most informative is challenging. Previous work has looked at word frequencies or semantic similarity, but this paper aims to systematically evaluate the diagnostic salience of specific morphosyntactic categories (like parts of speech) by using a novel masking approach with advanced language models.

Methods

The core of this research lies in a clever application of masked language models (MLMs), specifically AlephBERT, a pre-trained model for Hebrew. Instead of just using the model for classification, they employ a "masking" strategy:

  • Category Masking: For each experiment, they systematically mask all words belonging to a specific morphosyntactic category (e.g., all nouns). The MLM is then fine-tuned to classify the masked text into patient or control groups.
  • Control Masking: To isolate the effect of category masking, they also perform a control experiment where words are masked randomly with the same probability as the target category.
  • Comparison: The performance of the category-masked model is compared to the randomly-masked control. A significant drop in accuracy when a specific category is masked indicates that category's importance for classification.
  • Categories Examined: They focused on six categories: nouns, verbs, adverbs, adjectives, first-person words, and third-person words.

Datasets

The study used transcribed speech data from:

  • 23 male inpatients diagnosed with schizophrenia from the Be'er Yaakov-Ness Ziona Mental Health Center in Israel.
  • 26 healthy male controls recruited via social media.

All participants were native Hebrew speakers. The data was collected through interviews where participants described 14 images from the Thematic Appreciation Test (TAT) and answered four open-ended questions. This resulted in a total of 414 responses from patients and 468 from controls.

Key Insights and Ideas

  • Nouns are King (for diagnosis): The most striking finding is that masking nouns dramatically reduces classification accuracy. This suggests that the way nouns are used, rather than just their frequency, is a crucial differentiator.
  • Beyond Frequency: The paper emphasizes that simple word frequency doesn't explain the findings. While nouns are used frequently by both groups, adverbs, though less frequent, showed a significant difference in usage between patients and controls.
  • Imageability and Poverty of Speech: The authors hypothesize that nouns' imageability might make them easier to process, and that a focus on nouns in schizophrenia could be linked to "poverty of speech," where patients might rely more heavily on concrete nouns.
  • A Novel Evaluation Method: The masking approach itself is a significant contribution, offering a more nuanced way to assess the diagnostic salience of linguistic features compared to traditional frequency-based methods.

Key Results

  • Baseline Accuracy: The model achieved a baseline accuracy of 84.4% without any masking.
  • Noun Masking Impact: When nouns were masked, accuracy dropped significantly to 82.2% compared to the random masking control (84.6%), with a p-value < 0.0005. This was the most substantial drop observed.
  • Adverb Masking Impact: Masking adverbs also led to a decrease in accuracy (82.3% vs. 83.5% for random masking), with a p-value of 0.058, suggesting a less pronounced but still notable effect.
  • Other Categories: Masking verbs, adjectives, first-person, and third-person words did not result in statistically significant drops in accuracy compared to random masking.
  • Combined Masking: An additional experiment masking all words except nouns and adverbs yielded an accuracy of 82.8%, significantly outperforming the random masking control (75%), further supporting the importance of these two categories.

Limitations and Future Work

  • Participant Demographics: The study focused solely on male participants, limiting generalizability. The data was also collected from a specific mental health center, which might introduce site-specific biases.
  • Language Model Bias: The underlying AlephBERT model was trained on diverse, less controlled datasets, potentially introducing its own biases.
  • Explainability: While the study identifies nouns as important, the exact linguistic mechanisms (e.g., type of nouns used, their semantic relationships, or inflectional differences in Hebrew) remain unclear.
  • Future Directions: The authors plan to investigate the specific ways patients use nouns and how these differences relate to poverty of speech. They also aim to explore the source of the differences in adverb usage. Further work could also involve expanding the dataset to include female participants and a wider range of diagnostic criteria.

Summary updated

BibTeX
@inproceedings{shriki2022masking,
  author = {Yaara Shriki and Ido Ziv and Nachum Dershowitz and Eiran Harel and Kfir Bar},
  title = {{Masking Morphosyntactic Categories to Evaluate Salience for Schizophrenia Diagnosis}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2022},
}
Figure