TL;DR
This paper tackles the challenge of automatically triaging posts in online mental health forums to help moderators prioritize urgent cases. They developed a machine learning system that leverages not only the content of posts but also their surrounding context (like previous posts and user history), achieving first place in the CLPsych 2017 Shared Task.
What Problem Does This Paper Address?
Online peer-support forums for mental health are invaluable resources, but the sheer volume of posts makes it difficult for human moderators to identify and respond to critical situations, such as those involving suicide ideation. This can lead to delays in providing necessary support. The authors aim to automate this triage process, creating a system that can efficiently flag posts requiring immediate attention, thereby augmenting the capabilities of human moderators and ensuring timely intervention.
Methods
The core of the paper is a machine learning approach for post triage. They preprocess text by removing HTML, converting links and mentions to tokens, and standardizing text to lowercase. The novelty lies in the extensive feature engineering, which goes beyond just the post content. They extract a total of 2799 features, categorized into:
- Content-based features:
- Word2vec: Semantic representations of words and posts.
- N-grams: Unigrams and bigrams from post bodies.
- Metadata: Non-linguistic features like day of the week, time of day, board, author status (moderator or not), and thread creation time.
- Body: Features derived from psycholinguistic lexicons (e.g., for emotions, sentiment, personality traits), pronoun usage, and word counts.
- Context-based features:
- Interaction features: Derived from a directed mention graph, capturing relationships between posts and authors.
- Adjacent features: Features from the immediately preceding post in the same thread and the previous post by the same author.
- Author features: Aggregated features (mean values) of previous posts by the same author, providing a baseline of their typical behavior.
They employed Support Vector Machines (SVMs) with linear and RBF kernels, as well as ensemble methods (majority voting of SVMs). Feature selection was explored, and models were trained with class weights to handle imbalanced datasets.
Datasets
The study utilizes the dataset from the CLPsych 2017 Shared Task, which consists of posts from the Australian mental health forum Reachout.com.
- Training set: 157,963 posts written between July 2012 and March 2017. A subset of 1188 posts was labeled by three annotators for training.
- Testing set: 400 posts written between August 2016 and March 2017.
The dataset includes post text (subject and body) in XML format, along with metadata such as board, thread, post time, and author information.
Key Insights and Ideas
- Context is King (for flagging): The paper strongly suggests that incorporating contextual features (interactions, adjacent posts, author history) significantly improves the detection of "flagged" posts (those requiring moderator attention), especially for less severe but still important cases like amber-level posts.
- Content is Crucial (for urgency): Conversely, content-focused features tend to perform better in identifying the most urgent events (crisis and red levels).
- Overfitting is a Real Concern: Models with a large number of features, especially complex ones, can overfit to the training data, leading to reduced predictive power on unseen authors. This highlights the importance of diverse training data or online learning approaches.
- Feature Importance Reveals Nuances: The analysis of the most relevant features shows that crisis posts are associated with negative emotional states and specific pronoun usage, and importantly, that the content of interacting posts also provides valuable signals.
- Psycholinguistic Measures are Powerful: Features derived from psycholinguistic lexicons, particularly those related to well-being elements (PERMA), demonstrate significant predictive power.
Key Results
The authors' system achieved top performance in the CLPsych 2017 Shared Task:
- 1st place in the "flagged vs. non-flagged" f-score (using an ensemble of 7 SVM-RBF models). This metric is considered the most important by the organizers, as it measures the system's ability to identify posts needing moderator attention.
- 1st place in the "urgent vs. non-urgent" f-score (using an SVM-RBF trained with Word2vec + N-grams + subset of body features).
- 1st place in the "crisis vs. non-crisis" f-score (using an SVM-RBF trained with Word2vec + N-grams + subset of metadata features).
- 2nd position in the Macro-averaged f-score (using an ensemble of 4 SVM-linear models).
Notably, nine out of the top ten models in the "flagged vs. non-flagged" metric were from their team, underscoring the effectiveness of their approach.
Limitations and Future Work
- Overfitting: The authors acknowledge that complex models with many features can overfit to the training set, particularly concerning authors not seen in training. They suggest that training with larger, more diverse datasets or using online classifiers that continuously learn from moderator feedback could mitigate this.
- Feature Selection Impact: While they explored feature selection, it didn't yield significant performance improvements for SVM models, suggesting that the models might be robust enough to handle a larger feature set, or that the selection process could be further refined.
- Future Work: The paper implicitly suggests future work in exploring online learning to keep models up-to-date and in further investigating the predictive power of new psycholinguistic measures. They also highlight the potential of analyzing interactions among users and the content of interacting posts for improved triage.
Summary updated