TL;DR
This paper introduces ALBA, an adaptive language-based assessment system for mental health that dynamically selects questions and scores responses. ALBA, particularly its IRT-based variant (ALIRT), significantly reduces the number of questions needed to achieve high accuracy in assessing depression and anxiety, outperforming traditional fixed-question approaches.
What Problem Does This Paper Address?
Traditional mental health assessments, like the PHQ-9 for depression, rely on fixed sets of questions with rating scales. While effective, this approach can be inefficient, asking unnecessary questions and limiting the richness of information captured. Language-based assessments offer a more nuanced way to understand mental health by allowing individuals to express themselves freely. However, these can become lengthy and require substantial linguistic input for accurate scoring. This paper addresses the challenge of creating efficient, accurate, and adaptive language-based mental health assessments that can capture individual differences with minimal user effort.
Methods
The core of ALBA is its adaptive nature, which involves two key components: question ordering and response scoring. The authors propose two main methods:
- ALIRT (Adaptive Language-based IRT): This method combines Item Response Theory (IRT) with a supervised approach for polytomizing (categorizing) language responses. It uses word embeddings to represent responses and then applies IRT principles to select the most informative question at each step, aiming to maximize the information gained about the user's latent mental health state.
- Actor-Critic Model: Inspired by reinforcement learning, this method uses two models: a "Measure Model" that predicts the mental health score and an "Error Model" that predicts the error of the Measure Model for unadministered items. The Error Model then guides the selection of the next question to minimize predicted error.
Both methods are evaluated against several baselines, including random ordering and fixed strategies like Decision Trees, Forward Selection, and Backward Elimination.
Datasets
The study utilizes a dataset of open-ended language answers to eleven questions related to mental health and well-being, along with self-diagnostic tests (PHQ-9 for depression and GAD-7 for anxiety). Participants were recruited from Mechanical Turk and Prolific. The dataset consists of approximately 947 participants. The authors do not introduce a new benchmark dataset but rather use existing questionnaires and participant responses.
Key Insights and Ideas
- Adaptive is Better: Adaptive strategies consistently outperform fixed ordering methods, significantly reducing the number of questions needed for accurate assessment.
- ALIRT Shines: The ALIRT method, particularly when using IRT for scoring (ALIRT-L), demonstrates superior performance and scalability, achieving high correlations with latent trait scores with very few questions.
- Symptom Questions Less Informative Early On: For depression assessment, general mental health questions like "Describe Yourself" and "Describe Mental Health" are more informative in the initial stages than specific symptom-focused questions.
- IRT vs. CTT Scoring: While both IRT (latent estimate, L) and CTT (direct estimate, Ŷ) scoring paradigms can be used, IRT-based scoring generally yields better results, especially with adaptive strategies.
- Polytomization Trade-offs: While discretizing language responses (polytomization) is necessary for IRT, an optimal level exists (around 8-tomous) beyond which performance doesn't significantly improve and can even lead to issues like missing values.
Key Results
- ALIRT achieved a Pearson correlation of r > 0.9 with the latent depression score using just 3 questions, whereas traditional fixed ordering approaches needed at least 7 questions to reach a similar level of accuracy.
- The paper reports high correlations (Pearson r) between ALIRT's assessments and established depression (PHQ-9) and anxiety (GAD-7) scores, often exceeding 0.95 with a limited number of questions.
- The Actor-Critic model also showed improvements over baselines but was generally outperformed by ALIRT, despite having a significantly higher number of parameters.
Limitations and Future Work
- Self-Report Reliance: The study relies on self-reported questionnaires (PHQ-9, GAD-7), which may not always ensure complete reliability for diagnostic accuracy. Future work should validate assessments against clinical outcomes.
- English Language and Descriptive Words: The study was limited to English and descriptive words, not full open-ended texts. While the findings are expected to generalize, this is an area for future exploration.
- Dataset Size: The dataset, while sufficient for the proposed methods, is considered small, especially for training complex models.
- Future Directions: The authors envision ALBA as a step towards conversational diagnostic agents and suggest integrating adaptive testing into chatbot-style mental health assessments. They also highlight the potential for open sharing of data to advance research, while acknowledging privacy concerns.
Summary updated