TL;DR
This paper presents a framework for early detection of depression severity using social media posts, combining traditional Bag-of-Words (BoW) models with modern document embeddings. While the BoW approach with TF-IDF weighting showed promising results on a validation set, the overall performance on the test set was modest, highlighting the challenges of applying deep learning models to smaller, domain-specific datasets.
What Problem Does This Paper Address?
Early detection of mental illnesses like depression is crucial for timely intervention and preventing severe consequences. Social media platforms offer a rich, albeit noisy, source of data that can potentially reveal early signs of depression. However, accurately identifying the severity of depression from these posts is a complex task. This paper addresses the challenge of building a robust classification framework for this purpose, specifically for the DepSign-LT-EDI-ACL2022 shared task, which focused on classifying depression severity into three levels: no depression, moderate, and severe.
Methods
The authors explored two primary feature engineering techniques:
- Bag-of-Words (BoW) Model: This traditional approach represents text by the frequency of words (unigrams, bigrams, and trigrams) it contains. They experimented with two weighting schemes:
- Term Frequency-Inverse Document Frequency (TF-IDF): A standard method that weights terms based on their frequency in a document and their rarity across the corpus.
-
Entropy of Term Frequency: An alternative weighting scheme that considers the information content of a term. To handle the high dimensionality of BoW features, they employed chi-square statistics for feature selection.
-
Document Embeddings (Doc2Vec): This technique aims to capture the semantic meaning of entire documents by representing them as dense vectors. The Doc2Vec model was trained on the provided training corpus and data from a related eRisk shared task.
These features were then fed into several classifiers: Adaptive Boosting (AB), Logistic Regression (LR), Random Forest (RF), and Support Vector Machine (SVM).
Datasets
The primary dataset used was from the DepSign-LT-EDI-ACL2022 shared task, which consists of Reddit posts from users, labeled with depression severity (no depression, moderate, severe). This dataset was divided into training and validation sets for model development and evaluation, with a separate test set for final submission. The Doc2Vec model was also trained on data from the eRisk 2021 shared task for self-harm prediction.
Key Insights and Ideas
- BoW Still Holds Its Ground: Surprisingly, the conventional BoW model, particularly with TF-IDF weighting, outperformed more advanced document embedding techniques on the test data for this specific task. This suggests that for smaller, domain-specific datasets, the nuances captured by BoW might be more effective than general-purpose embeddings.
- Feature Combinations Matter: The results indicate that the combination of different feature types and classifiers is more important than relying on a single approach. Different models excelled with different feature sets.
- The Curse of Small Data for Deep Learning: The authors hypothesize that the underperformance of Doc2Vec might be due to the relatively small size of the training corpus. Deep learning models, including Doc2Vec, often require large amounts of data to learn robust semantic representations.
- Semantic Interpretation Challenges: The irregular and diverse nature of social media text poses a significant challenge for text classification, making it difficult for models to capture semantic meaning effectively.
Key Results
- The SVM classifier with TF-IDF weighted BoW features achieved the best performance on the validation set in terms of macro-averaged F-measure.
- On the test set, the submitted runs (which combined different feature types and classifiers) did not achieve top-ten rankings. Specifically, the best performing run achieved a macro-averaged F-measure of 0.438.
- The paper notes that the performance of the submitted models was "not reasonably well."
Limitations and Future Work
- Limited Data for Embeddings: The authors acknowledge that the Doc2Vec model's performance was likely hampered by the limited size of the training corpus.
- Lack of Clinical Expertise: The framework was developed without direct input from clinical experts, which could limit its ability to capture nuanced clinical indicators.
For future work, the authors propose:
- Developing pretrained transformer-based embeddings specifically for depression and mental disorders by leveraging larger datasets from social media, Wikipedia, and other relevant resources. This aims to improve the semantic understanding of the text.
- Exploring other feature engineering schemes and classifier combinations.
Summary updated