DepressionOne@LT-EDI-ACL2022: Using Machine Learning with SMOTE and Random UnderSampling to Detect Signs of Depression on Social Media Text.

Suman Dowlagar, Radhika MamidiACL 2022

TL;DR

This paper tackles the challenge of detecting depression from social media text, a task complicated by imbalanced datasets and long, complex posts. The authors propose a robust approach using SMOTE and random undersampling to balance the data, followed by an SVM classifier, achieving competitive results against neural network baselines.

What Problem Does This Paper Address?

Depression is a serious mental health issue, and early detection is crucial for effective treatment. Social media platforms offer a rich source of user-generated content where individuals often express their feelings and emotions. This paper focuses on leveraging this data to identify signs of depression, a task that presents several challenges:

  • Data Imbalance: Datasets for depression detection are often imbalanced, with a disproportionately large number of "not depressed" or "moderate" cases compared to "severe" cases. This imbalance can bias machine learning models, leading to poor performance on minority classes.
  • Long Text Sequences: Social media posts and comments can be lengthy, and traditional models like BERT, which have a token limit (e.g., 512 tokens), struggle to process these long sequences effectively.
  • Data Acquisition: Traditional methods for depression diagnosis often rely on clinical records or suicide notes, which are limited and difficult to obtain.

The LT-EDI-ACL2022 shared task aims to address these issues by providing a dataset of social media comments for classifying depression into three categories: "not depressed," "moderate," and "severe."

Methods

The core of the proposed method lies in addressing the data imbalance and selecting an appropriate classification model.

  • Data Resampling: To combat the imbalanced dataset, the authors employed a combination of SMOTE (Synthetic Minority Over-sampling Technique) and random undersampling.
  • Random Undersampling: This technique reduces the number of instances in the majority class (in this case, "moderate") by randomly removing them until a more balanced distribution is achieved.
  • SMOTE: This oversampling technique generates synthetic minority class samples. It works by selecting instances close in the feature space and creating new samples along the line connecting them.
  • The paper highlights that combining SMOTE with undersampling performs better than using either method alone.
  • Classification Model: After resampling, the authors opted for a Support Vector Machine (SVM) classifier. They chose SVM due to its robustness in handling text data and its ability to perform well even when dealing with long sequences, unlike models like BERT which have token limitations. The SVM was trained using TF-IDF (Term Frequency-Inverse Document Frequency) for text vectorization.

Datasets

The primary dataset used in this work is the one provided for the LT-EDI-ACL2022 shared task.

  • LT-EDI-ACL2022 Shared Task Dataset: This dataset consists of 16,632 comments in English. It is divided into:
  • Training set: 8,891 comments
    • Validation set: 4,496 comments
    • Test set: 3,245 comments

The dataset exhibits a class imbalance, with the "moderate" class having the most instances, followed by "not depression" and then "severe."

Key Insights and Ideas

  • Resampling is Key for Imbalance: The paper strongly suggests that effective handling of data imbalance through resampling techniques like SMOTE and random undersampling is crucial for achieving good performance in depression detection tasks.
  • SVM's Resilience to Long Texts: Despite the rise of transformer models like BERT, the study demonstrates that traditional models like SVM, when combined with appropriate text representation (TF-IDF), can still be highly effective, especially when dealing with long text sequences that pose challenges for BERT.
  • Trade-offs in Model Performance: While the proposed SVM approach showed improved performance on the "moderate" class, it led to a decrease in performance on the "not-depressed" class. This highlights the inherent trade-offs when balancing imbalanced datasets.

Key Results

The authors evaluated their submission against several baseline models, including SVM with TF-IDF, RNN, CNN, and BERT.

  • Overall Performance: Their submission, which combined SMOTE and random undersampling with an SVM classifier, achieved the highest macro-average F1-score of 0.60 and the highest overall accuracy of 0.65 among the tested models on the validation data.
  • Comparison with Baselines:
    • The SVM + SMOTE/undersampling approach outperformed the neural network models (RNN, CNN) in terms of macro-average F1-score and accuracy.
  • BERT, despite its power, did not perform as well, likely due to its limitations with long sequences in this specific dataset.
  • Confusion Matrix Analysis: The confusion matrices revealed that while the submission had better performance on "moderate" labels compared to the SVM + TF-IDF baseline, it showed a decreased performance on the "not-depressed" label. However, the number of correctly classified "moderate" instances was higher, contributing to the overall increased accuracy.

Limitations and Future Work

  • Limitations:
    • The paper acknowledges that the proposed model showed a decreased performance on the "not-depressed" class.
  • The BERT model's inability to effectively handle long sequences in this dataset is a noted limitation.
  • The authors mention that while resampling helps balance the data, it doesn't introduce entirely new information.
  • Future Work:
    • The authors express interest in testing "meta embedding models" on the given dataset in the future.
  • Exploring other techniques to better handle long sentences could be a valuable direction.

Summary updated

BibTeX
@inproceedings{dowlagar2022depressionone,
  author = {Suman Dowlagar and Radhika Mamidi},
  title = {{DepressionOne@LT-EDI-ACL2022: Using Machine Learning with SMOTE and Random UnderSampling to Detect Signs of Depression on Social Media Text.}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2022},
}
Figure