Semi-supervised CLPsych 2016 Shared Task System Submission

Nicolas Rey-Villamizar, Prasha Shrestha, Thamar Solorio, Farig Sadeque, Steven Bethard, Ted Pedersen

Introduction

Online mental health forums, such as ReachOut, offer critical peer support but often overwhelm human moderators. The primary challenge is automated triage: distinguishing between posts requiring immediate professional intervention (crisis or red) and those manageable by the community (amber or green). Technically, this task is hindered by a scarcity of expert-labeled data and a severe class imbalance, as urgent crisis events are rare compared to general discussion.

Methods

The proposed system utilizes a Multinomial Naive Bayes (MNB) classifier, enhanced by three core strategies:

  1. Psychologically Informed Features: In addition to standard n-grams and character affixes, the model incorporates the Linguistic Inquiry and Word Count (LIWC) lexicon. This allows the system to capture emotional tone (e.g., sadness, certainty) beyond literal vocabulary.
  2. Self-Training (Semi-supervised Learning): To mitigate the small-data bottleneck, the initial model predicted labels for over 63,000 unlabeled posts. The 100 most confident "pseudo-labeled" samples were added back to the training set, expanding the model's exposure to diverse vocabulary.
  3. Bayesian Optimization: Hyperparameters, including smoothing terms and class weights, were tuned using the SigOpt framework to systematically optimize performance.

Datasets

The study utilized the CLPsych 2016 Shared Task dataset, sourced from au.reachout.com. The data distribution highlights the extreme imbalance, with "green" posts forming the majority and "crisis" posts being exceptionally rare.

Distribution of training, test, and unlabeled datasets.

Key Results

The system's performance was evaluated using a macro-averaged F-score across the urgent categories (crisis, red, amber). The final model achieved an official score of 0.34, placing it slightly above the median of all participating teams.

Summary of system performance and baseline comparisons.

The model demonstrated a significant improvement (0.12 increase in macro F1) through self-training, underscoring the value of unlabeled data. While it performed well on "amber" posts (F1: 0.58), it failed to identify the single "crisis" post in the test set. This failure was attributed to "unseen vocabulary"—the specific crisis post used terms not present in the training data, a known limitation of count-based models.

Discussion and Future Work

The skewed data distribution remains the primary obstacle to reliable triage in life-or-death scenarios. Future research should explore word embeddings (e.g., Word2Vec) to better handle semantic similarity and synonyms. Additionally, incorporating longitudinal patterns—analyzing a user's posting history over time—could provide more context than treating posts as isolated events.

Summary updated

BibTeX
@inproceedings{reyvillamizar2016semi,
  author = {Nicolas Rey-Villamizar and Prasha Shrestha and Thamar Solorio and Farig Sadeque and Steven Bethard and Ted Pedersen},
  title = {{Semi-supervised CLPsych 2016 Shared Task System Submission}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2016},
  doi = {10.18653/v1/w16-0322},
}
Figure