TL;DR
This paper tackles the challenge of detecting mood shifts and predicting suicide risk from imbalanced social media data. They propose a BERT-based approach for mood change detection and a Random Forest model with novel features for suicide risk prediction, achieving promising results on the CLPsych shared task.
What Problem Does This Paper Address?
Social media platforms offer a rich, albeit noisy, source of data for understanding mental health. Researchers are increasingly interested in leveraging this data to identify subtle shifts in users' moods over time and to predict the severity of suicide risk. However, two major challenges hinder progress: 1) the inherent imbalance in social media data, where certain events or moods are far less frequent than others, and 2) the need for models that can effectively capture temporal dynamics and extract meaningful signals from user-generated text. This paper specifically addresses these issues within the context of the CLPsych shared task, focusing on predicting mood changes (Subtask A) and suicide risk levels (Subtask B) from Reddit data.
Methods
For Subtask A (Mood Change Detection), the authors explored several approaches. They preprocessed the data by cleaning special characters, URLs, and standardizing word boundaries. To combat the severe class imbalance (where "no change" is overwhelmingly dominant), they employed undersampling, specifically removing posts with a positive sentiment (polarity >= 0.2) that were deemed less indicative of mood shifts. They then experimented with BERT for post-level classification and LSTM models (with GloVe and SpaCy embeddings) to capture sequential information.
For Subtask B (Suicide Risk Prediction), the focus shifted to user-level classification. Recognizing the limited dataset size, they opted against undersampling and instead focused on extracting extra features to enhance model performance. These features included:
- Sentiment: Aggregated sentiment of all posts for a user.
- Polarity: The overall polarity score across a user's posts.
- Number of Suicidal Words: A count of words from a manually curated list of suicide-related terms found in a user's posts.
They then applied Random Forests (RF), testing three variants: one with word embeddings, one with word embeddings and the extracted features, and one with only the extracted features (RF3).
Datasets
The study utilized data from Reddit, specifically focusing on subreddits related to mental health for Subtask A and the "r/SuicideWatch" subreddit for Subtask B.
- Subtask A: Data from 186 users, comprising 256 timelines and 6205 posts, with an average user timeline of 2 months. The data was annotated into three labels: "IS" (mood shifts), "IE" (gradual mood progression), and "0" (no change). The dataset was severely imbalanced, with 79% for "0", 15% for "IE", and 6% for "IS".
- Subtask B: Data from 127 users, annotated into four suicide risk categories: "None", "Low", "Moderate", and "Severe". For modeling, "Low" and "None" were merged, resulting in three classes.
Key Insights and Ideas
- BERT's Power for Mood Shifts: The BERT model, even without explicit sequential modeling of posts, outperformed LSTM models for Subtask A. This suggests BERT's strong ability to capture nuanced semantic information within individual posts is highly effective for detecting mood changes.
- Undersampling's Effectiveness: The undersampling strategy for Subtask A, while reducing data, significantly improved the model's ability to detect the underrepresented "IS" class, leading to better precision and recall for this crucial category.
- Feature Engineering for Suicide Risk: For Subtask B, the simple yet effective feature of counting suicidal words (RF3) surprisingly outperformed models that included word embeddings. This highlights the power of domain-specific feature engineering when dealing with imbalanced datasets and sensitive topics.
- The Value of Simplicity: The success of RF3 in Subtask B suggests that for certain tasks, especially with limited data, carefully engineered features can be more impactful than complex embedding models.
Key Results
- Subtask A: The BERT model achieved the best performance across most evaluation metrics for Subtask A, outperforming baseline models and LSTM variants in both post-level and coverage-based metrics. Specifically, BERT achieved a macro-average F1 score of 0.692 for coverage-based metrics and 0.871 for post-level metrics.
- Subtask B: The Random Forest model using only extracted features (RF3) achieved the highest accuracy (0.516) for Subtask B, notably being the only model capable of predicting the "Low" suicide risk class. This model achieved an F1 score of 0.638 for the "Low" class.
Limitations and Future Work
The authors acknowledge several limitations:
- Suicidal Words List: The list of suicidal words was compiled from various sources and manual inspection; it could be expanded and validated further.
- Embedding Models: They did not fine-tune embedding models (except BERT) to their specific dataset, relying on general pre-trained embeddings.
- Oversampling: Due to time constraints, they could not explore oversampling techniques to address data imbalance.
Future work suggestions include:
- Exploring oversampling techniques (e.g., using GPT-3) to balance datasets.
- Expanding and validating the suicidal words list.
- Extracting additional meaningful features from text to further enhance classification results.
Summary updated