TL;DR
This paper details the USI-UPF team's approach to the CLPsych 2019 shared task, focusing on assessing suicide risk from social media posts. They developed machine learning models using a combination of lexicon-based features, word embeddings, n-grams, and statistical measures, achieving competitive results by integrating these diverse feature sets.
What Problem Does This Paper Address?
Suicide remains a significant public health concern globally, with a particularly high mortality rate among young adults. While the internet has unfortunately facilitated access to harmful information related to suicide, it also presents an opportunity for early detection and intervention. This paper addresses the challenge of automatically identifying individuals at risk of suicide by analyzing their online social media activity. The goal is to develop computational methods that can assist mental health professionals in detecting at-risk individuals, ultimately contributing to suicide prevention efforts.
Methods
The USI-UPF team employed a multi-faceted approach to suicide risk assessment, leveraging a range of machine learning techniques and feature engineering strategies.
- Feature Engineering:
- Bag of Words and N-grams: They utilized TF-IDF vectorization for unigrams up to 5-grams at the word level. Stop words were retained as they can be indicative of emotional state and personality. N-grams appearing in fewer than five documents were removed to reduce feature space.
- Word Embeddings: Pre-trained GloVe embeddings (200 dimensions) were used, with document representations obtained by averaging word embeddings column-wise.
- Lexicon-based Features: They incorporated word frequencies from categories within the LIWC2007 dictionary, including a list of antidepressants and "absolutist" words, which have been linked to anxiety, depression, and suicidal ideation.
- Statistical Features: Features such as the total number of posts per user, average post size, and the number of subreddits a user posted in were included.
- Feature Selection: A chi-square test was employed to select the most predictive n-grams for each risk class, helping to reduce the feature space and improve model performance.
- Classifiers: The team used a One-vs-All (OVA) strategy for multi-class classification. They experimented with Logistic Regression (LR) and Linear Support Vector Machines (SVM), with L2 regularization to prevent overfitting.
Datasets
The primary dataset used was the University of Maryland Reddit Suicidality Dataset. This corpus consists of posts from Reddit, including:
- Posts from users who participated in the "Suicide Watch" (SW) forum (positive instances of suicidality).
- Posts from a comparable number of control users who did not post on SW.
A subset of SW users were manually labeled by human annotators on a four-point scale:
- (a) No Risk
- (b) Low Risk
- (c) Moderate Risk
- (d) Severe Risk
The dataset comprised 993 users for training and 248 users for testing.
Key Insights and Ideas
- The Power of Combined Features: The most effective models were achieved by integrating lexicon-based features, a carefully selected set of n-grams, and statistical measures. This highlights that no single feature type is sufficient for robust suicide risk assessment.
- Value of Function Words: Retaining stop words (function words) proved beneficial, as they can offer insights into a user's emotional state, personality, and thinking style.
- Distinctive Vocabulary for Risk Levels: Users in the "Severe Risk" category exhibited a vocabulary quite distinct from other users, suggesting that specific linguistic patterns are associated with higher risk levels.
- Impact of Domain-Specific Data: Including posts from the "Suicide Watch" forum significantly improved performance, particularly for identifying lower-risk users, but also introduced some noise for higher-risk categories.
- Interpretability of OVA: The One-vs-All strategy offered interpretability, allowing for the inspection of individual classifiers for each risk class.
Key Results
The paper reports results on the CLPsych 2019 shared task, with the best-performing models being combinations of lexicon-based features, selected n-grams, and statistical measures. While specific overall accuracy scores are not explicitly stated as a single metric, the paper presents detailed Precision, Recall, and F1 scores per class for their top models in Table 2. For instance, one of their top models for Task A (LR_Reduced_LIWC_Stats) achieved a macro-average F1 of 28.10% on the test set. Table 3 shows the macro-average F1 for their selected models across training and test sets, with rankings indicating their performance relative to other participants (e.g., 11th for Task A, 6th for Task B, 6th for Task C). They noted that class 'b' (Low Risk) was particularly challenging to predict due to fewer training samples and similar vocabulary to 'No Risk' users.
Limitations and Future Work
- Class Imbalance and Overlap: The authors acknowledge that predicting "Low Risk" (class b) is difficult, partly due to a lower number of training samples and its linguistic similarity to "No Risk" users.
- Noise from Domain-Specific Data: While beneficial, the inclusion of data from specific forums like "Suicide Watch" can introduce noise for other risk categories.
- Future Work: The paper doesn't explicitly detail future work, but the insights gained suggest avenues for further research, such as exploring more sophisticated feature interaction methods, investigating other embedding techniques, and potentially developing more nuanced risk stratification models. The observed differences in vocabulary across risk levels also point towards the potential for more targeted linguistic analysis.
Summary updated