TL;DR
This paper presents a system developed by GW and UMD for the CLPsych 2016 Shared Task, aiming to classify online posts from teens and young adults into different levels of urgency for mental health support. They achieved a strong performance using a two-stage ensemble classifier that combines lexical and non-lexical features, demonstrating the effectiveness of a multi-faceted approach for this challenging task.
What Problem Does This Paper Address?
The paper tackles the critical problem of identifying individuals in distress online, specifically focusing on posts from teens and young adults seeking help on the ReachOut platform. Suicide is a leading cause of death for young people, and a significant percentage experience hopelessness, bullying, and suicidal ideation. The challenge lies in automatically classifying these posts into categories of urgency (green, amber, red, crisis) to ensure timely intervention by moderators. This is particularly difficult due to the imbalanced nature of the dataset, with very few "crisis" posts compared to others, and the subtle linguistic cues that indicate severe distress.
Methods
The GW/UMD system employs a supervised learning approach for classification. They utilize Stanford CoreNLP for preprocessing tasks like tokenization and lemmatization. Their feature set is diverse, encompassing:
- Lexical Features: TF-IDF weighted unigrams (most important words), part-of-speech (POS) tag counts (adverbs, pronouns, modal auxiliaries), and linguistic inquiry and word count (LIWC) categories.
- Sentiment Features: Counts of negative, positive, and neutral sentences within a post, derived from Stanford CoreNLP's sentiment analysis.
- Non-Lexical Features:
- ReachOut Meta-data: Number of views, time of day (bucketed into 3-hour windows), and the specific forum board the post appeared on.
- Emotion Features: Counts of emotion words from the NRC Word-Emotion Association Lexicon (anger, anticipation, disgust, fear, joy, sadness, surprise, trust).
- User-Specific Features: Explicit user mentions (@ mentions) and word count.
- Additional Explored Features (post-submission): ReachOut author status (moderator/staff), mental disease lexicon matches, word shape (repeated characters, all caps), and word embeddings (averaged word2vec vectors).
The core of their system is a two-stage classifier. The first stage uses a Support Vector Machine (SVM) with lexical features (TF-IDF weighted unigrams). The second stage is a logistic regression classifier that takes the output probabilities from the SVM and combines them with all other non-lexical features. They also experimented with individual classifiers (uniLR, uniSVM, nonLexLR) and a majority vote ensemble (maj1).
Datasets
The primary dataset used is from the CLPsych 2016 Shared Task, consisting of posts from the ReachOut platform. The dataset is annotated with four classes: green, amber, red, and crisis. The paper notes that the dataset is highly imbalanced, with a particularly small number of "crisis" and "red" posts.
Key Insights and Ideas
- Ensemble Methods Shine: The paper highlights that ensemble methods, particularly their two-stage approach, are effective, especially when dealing with small and imbalanced training data. This is crucial for real-world applications where perfect data representation is rare.
- Feature Diversity is Key: Combining a wide range of features—from simple word counts to sentiment, emotions, and even metadata like posting time—significantly improves performance. No single feature type appears to be a silver bullet.
- Mental Disease Lexicon's Impact: While not part of their official submission, the exploration of a mental disease lexicon revealed its ability to capture the single crisis instance in the test data and improve recall for "red" posts. This suggests that domain-specific lexicons can be highly valuable.
- LIWC's Consistent Value: LIWC features consistently provided a boost in performance, aligning with previous research on their utility in psychological text analysis.
Key Results
The best official submission, the 2stage classifier, achieved a macro F1-score of 0.36. While this was 6% short of the best system in the shared task, further experimentation after the task yielded an improved macro F1 score of 0.45. This post-submission result was significantly boosted by the inclusion of the mental disease lexicon feature. The paper also shows that a non-lexical classifier (nonLexLR) performed equally well to a single classifier using a combination of lexical and non-lexical features, suggesting the power of non-lexical cues.
Limitations and Future Work
The authors acknowledge the highly imbalanced nature of the dataset and the small number of "crisis" and "red" posts as key challenges. They also note that their official submission was 6% short of the best system.
For future work, they plan to:
- Experiment with an extended ReachOut meta-data feature set.
- Expand LIWC features by incorporating word embeddings.
Summary updated