WWBP-SQT-lite: Multi-level Models and Difference Embeddings for Moments of Change Identification in Mental Health Forums

Adithya V Ganesan, Vasudha Varadarajan, Juhi Mittal, Shashanka Subrahmanya, Matthew Matero, Nikita Soni, Sharath Chandra Guntuku, Johannes Eichstaedt, H. Andrew SchwartzNAACL 2022

TL;DR

This paper introduces two novel approaches, difference embeddings and multi-level recurrent transformers, for detecting subtle shifts in mental health states within online forum posts. By analyzing sequential text data, the models aim to identify moments of mood change, offering a more dynamic understanding of mental well-being than static analyses.

What Problem Does This Paper Address?

Understanding and monitoring mental health at scale is a significant challenge. Traditional methods often rely on static data, like single diagnostic labels or infrequent clinical visits, which fail to capture the dynamic nature of mental states. Online forums, however, provide a rich source of longitudinal data that can reveal subtle changes in an individual's mood and mental well-being over time. The problem this paper addresses is how to effectively detect and classify these "moments of change" in user-generated text, moving beyond static assessments to a more nuanced, temporal understanding of mental health. This is crucial for early detection of potential issues and timely intervention.

Methods

The authors explore two primary methods for capturing psychological changes in text:

  • Difference Embeddings: This technique leverages pre-trained language models (specifically RoBERTa) to generate embeddings for each message. The core idea is to compute the difference between the embedding of a current message and the embedding of the preceding message. This difference vector is hypothesized to capture the change in sentiment or topic. To handle the high dimensionality of these embeddings, they employ Principal Component Analysis (PCA) for dimensionality reduction. Several variations are tested, including using only the current message, concatenating current and previous messages, and using the difference vector alone or in combination.
  • Human-aware Recurrent Transformers (HaRT): This approach builds upon transformer architectures (specifically GPT2) to create message representations that also encode the author's context and history. The HaRT model is designed to capture latent representations of the author, allowing for a more personalized understanding of their language use over time. The authors experiment with fine-tuning different layers of the HaRT model.

Datasets

The study utilizes data from the CLPsych 2022 shared tasks, specifically focusing on Task A (modeling user state changes over time). The training data consists of 5,143 Reddit posts from 149 users across 204 timelines. These posts were annotated with three classes of mood change: "Ordinary" (no significant change), "In Switch" (positive to negative or vice versa), and "In Escalation" (increasing intensity). The data was further split into internal training and validation sets. For Task B (suicide risk prediction), a subset of this data was used, excluding posts annotated as "N/A".

Key Insights and Ideas

  • The power of sequential analysis: The paper strongly suggests that analyzing text sequentially, rather than in isolation, is key to understanding dynamic mental health states. Capturing the change between messages is more informative than just the content of individual messages.
  • Difference embeddings as a strong baseline: The simple yet effective difference embedding approach, especially when applied to dimension-reduced RoBERTa embeddings, proved to be a highly competitive method for detecting mood changes, particularly for "switches."
  • HaRT's advantage in temporal context: The HaRT model, by incorporating author history, shows promise in capturing temporal dynamics, outperforming simpler transformer models in some aspects.
  • Theoretical lexica (HypLex) are relevant: Pre-defined lexica related to psychological constructs (like depression, anxiety, loneliness) showed significant associations with mood change patterns, highlighting the link between linguistic expression and mental state.
  • Switches are easier to detect than escalations: The models found it easier to identify abrupt mood shifts ("In Switch") compared to gradual increases in intensity ("In Escalation"). This is likely due to the more subtle nature of gradual changes, making them harder to distinguish from "ordinary" states.

Key Results

  • For Task A (mood change identification), the difference-between-embeddings technique applied to RoBERTa embeddings achieved the highest overall F1 score of 0.61.
  • This difference embedding method particularly outperformed other baselines in detecting "switches" (F1 = 0.33) and "escalations" (F1 = 0.61).
  • The HaRT model (CLS variant) performed better than standard RoBERTa embeddings, suggesting the importance of encoding author history.
  • Theoretical lexica (HypLex) alone achieved a macro F1 of 0.44 on the internal validation set for Task A.
  • Language use patterns associated with constructs like depression, stress, anger, and anxiety were predictive of mood switches and escalations.

Limitations and Future Work

  • Data limitations: The study notes that the "lite" version of their system was constrained by limited data availability (only 4 days of data), impacting the full exploration of their planned human-level and temporal approaches.
  • Difficulty with gradual changes: The difference embedding approach struggled to capture "escalations" (gradual mood changes), often misclassifying them as "ordinary" states. This is attributed to the subtle nature of these changes.
  • Hyperparameter tuning: Due to time constraints, hyperparameter values were largely based on reported values from previous work, potentially limiting optimal performance.
  • Future work: The authors suggest further exploration of HaRT variants and more sophisticated methods for capturing gradual mood changes. They also highlight the potential of combining different representation types (e.g., embeddings, topics, lexica) for improved performance. The ethical implications of deploying such technologies are also emphasized, stressing the need for collaboration with clinical practitioners.

Summary updated

BibTeX
@inproceedings{ganesan2022wwbp,
  author = {Adithya V Ganesan and Vasudha Varadarajan and Juhi Mittal and Shashanka Subrahmanya and Matthew Matero and Nikita Soni and Sharath Chandra Guntuku and Johannes Eichstaedt and H. Andrew Schwartz},
  title = {{WWBP-SQT-lite: Multi-level Models and Difference Embeddings for Moments of Change Identification in Mental Health Forums}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2022},
}
Figure