Explaining Models of Mental Health via Clinically Grounded Auxiliary Tasks

Ayah Zirikly, Mark DredzeNAACL 2022

TL;DR

This paper tackles the challenge of making AI models for mental health diagnosis more understandable to clinicians. They propose a multi-task learning approach that uses clinically relevant "auxiliary tasks" (predicting specific depression symptoms) to improve the explainability of the main depression detection model, leading to more meaningful explanations.

What Problem Does This Paper Address?

Diagnosing mental health conditions, particularly depression, is crucial but complex. While Natural Language Processing (NLP) models can analyze text (like social media posts) to detect signs of depression, these models often act as "black boxes." This lack of transparency makes it difficult for clinicians, who rely on structured assessment frameworks, to trust and integrate these AI tools into their practice. Clinicians need to understand why a model flags someone as depressed to make informed decisions about diagnosis and treatment. Existing explainability techniques (like LIME and SHAP) can highlight important words, but the meaningfulness of these explanations is limited if the underlying model isn't grounded in clinical understanding.

Methods

The core of this work is a multi-task learning (MTL) framework. Instead of just training a model to predict if a tweet indicates depression, they train it to simultaneously predict:

  1. Depression: The primary task.
  2. PHQ-9 Categories (Symptoms): Nine auxiliary tasks, each corresponding to a symptom measured by the widely used Patient Health Questionnaire-9 (PHQ-9).

They compare this MTL approach against two single-task models:

  • Logistic Regression (LR): A simpler, interpretable model.
  • Multilayer Perceptron (MLP): A more complex neural network.

For explaining the models, they use LIME (Local Interpretable Model-agnostic Explanations). LIME generates explanations by creating local, interpretable models around individual predictions. The key innovation here is comparing LIME explanations generated from:

  • The depression task alone (D).
  • The symptom prediction tasks alone (S/S-comb).
  • The combined depression and symptom tasks (D+S).

Datasets

The primary dataset used is the Depression to (2) Symptoms (D2S) dataset. This dataset consists of English tweets, originally collected and annotated for depression and specific PHQ-9 symptoms. The authors filtered it down to 2132 depressive tweets and 5698 control tweets due to some unavailability. They also created a new manually labeled "D2S-explain" dataset by sampling 105 depressive tweets from the test set and having an expert annotator identify the most relevant tokens for explaining depression.

Key Insights and Ideas

  • Clinically Grounded Explanations are Better: The central hypothesis is that by incorporating clinically meaningful auxiliary tasks (like predicting PHQ-9 symptoms), the explanations generated for depression detection become more relevant and useful to clinicians.
  • Auxiliary Tasks Aid Interpretability: Even if the auxiliary tasks themselves aren't perfectly predicted, their inclusion in the model training process can lead to better explanations for the primary depression task.
  • MTL Improves Symptom Prediction: The MTL model showed significantly better performance on the symptom prediction tasks, especially for less frequent symptoms, compared to single-task models, suggesting it better handles imbalanced data.
  • Partial Matches Matter for Explanations: For evaluating explanations, a partial match between a generated token and a gold annotation is considered a "true positive" if it highlights clinically relevant information, acknowledging that clinicians can infer meaning from partial cues.

Key Results

  • Explanation Recall: In scenario (D) (depression task only), the LR model achieved the highest explanation recall (0.61), likely due to its better performance on the depression task itself.
  • Augmenting Explanations: When augmenting explanations with symptom information (scenario D+S):
  • LR saw a 1.9% increase in explanation recall.
  • MLP and MTL saw smaller increases (around 1%).
  • The MTL model, which already incorporates symptom information internally, showed the highest recall in the D+S scenario (0.533), suggesting its internal representation of symptoms contributes to better explanations.
  • MTL Performance: The MTL model outperformed single-task models on symptom classification, particularly for less frequent symptoms, demonstrating its robustness to data imbalance.

Limitations and Future Work

  • Dataset Limitations: The D2S dataset is from Twitter, which has inherent reliability concerns. The PHQ-9 annotations are binary, not the nuanced 4-point scale, which can degrade annotation quality. The dataset is also small and imbalanced.
  • Annotation Proxy: The manually annotated explanations are a proxy for what a clinician might find informative; true clinical evaluation is needed.
  • Future Work: The authors hope for larger, more clinically representative datasets with richer PHQ-9 annotations. They also suggest further exploring how to best integrate clinical practices into model design and evaluating explanations in a real clinical setting.

Summary updated

BibTeX
@inproceedings{zirikly2022explaining,
  author = {Ayah Zirikly and Mark Dredze},
  title = {{Explaining Models of Mental Health via Clinically Grounded Auxiliary Tasks}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2022},
}
Figure