Analyzing the use of existing systems for the CLPsych 2019 Shared Task

Alejandro González Hevia, Rebeca Cerezo Menéndez, Daniel Gayo-AvelloNAACL 2019

TL;DR

This paper explores reusing previously trained systems for suicide risk detection on a new CLPsych shared task dataset. While pre-trained systems showed slightly lower performance than a baseline trained on the new data, they still performed reasonably well, highlighting the potential of transfer learning in this sensitive domain. The study also investigates the utility of cross-lingual word embeddings for this task.

What Problem Does This Paper Address?

The paper tackles the challenge of accurately predicting suicide risk from online user posts, a critical task for mental health support. Specifically, it addresses whether systems trained on data from a previous CLPsych shared task (CLPsych 2016) can be effectively repurposed for a new, similar task (CLPsych 2019). This is important because creating and labeling large datasets for suicide risk detection is time-consuming and ethically sensitive. The authors aim to see if existing models can generalize to new data, potentially reducing the need for extensive re-training and data collection. They also investigate the use of cross-lingual word embeddings to see if they can improve performance, especially in scenarios where data might be available in multiple languages.

Methods

The researchers employed three distinct systems for the CLPsych 2019 Shared Task:

  • pretrained_svm: This system utilized a Support Vector Machine (SVM) trained on the ReachOut dataset (from CLPsych 2016). It combined TF-IDF features, document embeddings (generated using doc2vec), and VAD (Valence, Arousal, Dominance) scores derived from a lexicon.
  • pretrained_rnn: This system used a Recurrent Neural Network (RNN) with Gated Recurrent Units (GRUs), trained on the ReachOut data. The primary features for this model were cross-lingual word embeddings, aiming to capture nuances across languages.
  • custom_svm: This served as a baseline system, an SVM trained directly on the CLPsych 2019 SuicideWatch dataset. It used TF-IDF features and VAD scores.

All systems focused solely on the text content of posts, excluding metadata. Text preprocessing involved removing HTML characters, stopwords, tokenizing, and lowercasing. Hyperparameters for the models were tuned using 5-fold cross-validation.

Datasets

The study utilized two primary datasets:

  • CLPsych 2019 Shared Task Data (SuicideWatch subreddit): This dataset, adapted from the University of Maryland Reddit Suicidality Dataset, contained 847 labeled posts from 496 users. Labels were categorized into four levels of suicide risk: No risk, Low risk, Moderate risk, and Severe risk.
  • CLPsych 2016 Shared Task Data (ReachOut forum): This dataset comprised 65,024 posts from an Australian mental health forum for young people, with 1,227 labeled posts. The labels (Green, Amber, Red, Crisis) indicated the level of attention required from a mental health professional.

The paper notes that while the labels are conceptually similar, the annotation criteria and focus differed between the two datasets (user risk level vs. professional attention needed), and the ReachOut labels were post-level while SuicideWatch labels were user-level.

Key Insights and Ideas

  • Transfer Learning Potential: Reusing systems trained on older, related datasets (ReachOut) for a new task (SuicideWatch) is feasible and yields reasonable results, even with differing annotation schemes. This suggests that models can learn generalizable patterns for suicide risk detection.
  • Dataset Differences Matter: The performance gap between pre-trained systems and the custom baseline highlights the impact of dataset-specific labeling criteria and data distributions. The imbalance in label frequencies between the two datasets (e.g., "Severe Risk" being most frequent in SuicideWatch but least in ReachOut) likely contributed to this.
  • Cross-Lingual Embeddings' Role: The pretrained_rnn system, utilizing cross-lingual word embeddings, performed comparably to the pretrained_svm system. This indicates that these embeddings can be a viable option for tasks involving multilingual data or when aiming for broader applicability.
  • Task Granularity: The systems struggled with fine-grained risk distinctions (Low, Moderate, Severe) but performed better when classifying posts into broader categories like "flagged" (risk vs. no risk) or "urgent" (moderate/severe vs. low/no risk). This suggests that the task of precise risk level prediction is more challenging than general risk identification.

Key Results

  • The custom_svm (baseline trained on CLPsych 2019 data) outperformed the pre-trained systems in both accuracy and F1-score on the training data (0.61 accuracy, 0.32 F1).
  • On the test set, the custom_svm achieved an accuracy of 0.51 and an F1-score of 0.31. The pretrained_svm achieved 0.49 accuracy and 0.27 F1, while pretrained_rnn achieved 0.52 accuracy and 0.30 F1.
  • Despite lower macro-averaged F1 scores, the systems showed strong performance on binary classifications:
  • "Flagged" (risk vs. no risk) F1 scores ranged from 0.79 to 0.89.
  • "Urgent" (moderate/severe vs. low/no risk) F1 scores ranged from 0.84 to 0.88.

Limitations and Future Work

  • Annotation Differences: The authors acknowledge that differences in annotation instructions and criteria between the CLPsych 2016 and 2019 tasks are a primary reason for the performance decrease in pre-trained systems.
  • Imbalanced Datasets: Both datasets exhibit label imbalance, which can pose challenges for model training and evaluation.
  • Future Work: The paper suggests further exploration of cross-lingual word embeddings, as they offer potential for handling data from various languages and forums. Investigating how to better bridge the gap caused by differing annotation schemes in future transfer learning efforts would also be valuable.

Summary updated

BibTeX
@inproceedings{hevia2019analyzing,
  author = {Alejandro González Hevia and Rebeca Cerezo Menéndez and Daniel Gayo-Avello},
  title = {{Analyzing the use of existing systems for the CLPsych 2019 Shared Task}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2019},
}
Figure