AraDepSu: Detecting Depression and Suicidal Ideation in Arabic Tweets Using Transformers

Mariam Hassib, Nancy Hossam, Jolie Sameh, Marwan TorkiWANLP @ EMNLP 2022 · OpenReview

TL;DR

This paper introduces AraDepSu, a novel dataset and methodology for detecting depression and suicidal ideation in Arabic tweets using Transformer models. They demonstrate that MARBERT, a model specifically trained on Arabic dialects and tweets, significantly outperforms other models, achieving 91.20% accuracy. This work is crucial for addressing the under-resourced area of mental health detection in Arabic social media.

What Problem Does This Paper Address?

Mental health, particularly depression and suicidal ideation, is a significant global concern. In the Middle East, and specifically Egypt, suicide rates are alarmingly high, yet there's a notable lack of awareness and stigma surrounding mental health issues. Social media platforms like Twitter have become a space where individuals express their emotions and thoughts, offering a potential avenue for early detection of mental health struggles. However, existing research and tools for detecting depression and suicidal ideation are heavily skewed towards English, leaving a critical gap for Arabic-speaking populations. The complexity of Arabic dialects further complicates the development of effective NLP models for this region. This paper aims to bridge this gap by developing a robust system for identifying depression and suicidal ideation in Arabic tweets.

Methods

The researchers employed a multi-stage approach:

  • Dataset Creation (AraDepSu): They scraped over 10,000 Arabic tweets using keywords related to depression and suicidal ideation, spanning from 2016 to 2022. To enhance the dataset's diversity and represent real-world usage, they also incorporated 1,230 records from an existing Arabic mood-changing and depression dataset. Crucially, they introduced a "neutral" class to better capture the nuances of social media expression, resulting in a three-class dataset: "depressed," "suicidal," and "neutral."
  • Data Pre-processing: Tweets were cleaned by removing irrelevant text, such as hyperlinks, and duplicate records.
  • Manual Labeling: The dataset was manually labeled by annotators following specific instructions. Each tweet was assigned one of the three classes. Disagreements were resolved by prioritizing the authors' decisions.
  • Model Training and Evaluation: They trained over 30 different Transformer-based models on their newly created AraDepSu dataset. The models were evaluated using standard metrics like accuracy, precision, recall, and F1-score.

Datasets

  • AraDepSu: A newly created dataset consisting of 20,213 manually labeled Arabic tweets, categorized into "depressed," "suicidal," and "neutral." This dataset is a key contribution, addressing the scarcity of Arabic-language mental health data.
  • Existing Arabic Datasets: The paper also mentions incorporating data from the "Modern Standard Arabic mood changing and depression dataset" (Maghraby and Ali, 2022) and references other Arabic sentiment analysis datasets (e.g., ArTwitter, TEAD, BRAD, ASTD) in Table 1, highlighting the broader landscape of Arabic NLP resources.

Key Insights and Ideas

  • The Power of Dialect-Specific Models: The study strongly suggests that models trained on Arabic dialects, like MARBERT, significantly outperform those trained on Modern Standard Arabic (MSA) or general multilingual models when dealing with real-world Arabic tweets. This is because social media content often uses informal, dialectal language.
  • MARBERT's Superiority: MARBERT, a model specifically designed for Arabic dialects and tweets, emerged as the top performer. This highlights the importance of domain-specific pre-training for nuanced tasks like mental health detection.
  • Nuance in Social Media Expression: The inclusion of a "neutral" class was vital. It acknowledges that not all expressions of sadness or hardship necessarily indicate clinical depression or suicidal ideation, allowing for a more accurate classification.
  • Stigma and Awareness Gap: The paper underscores the cultural context of the Middle East, where mental health stigma is a significant barrier, making early detection through social media even more critical.

Key Results

The best-performing model, MARBERT, achieved impressive results on the AraDepSu dataset:

  • Accuracy: 91.20%
  • Macro-Average Precision: 88.74%
  • Macro-Average Recall: 88.50%
  • Macro-Average F1-Score: 88.75%

The study also found that models pre-trained on tweets (like MARBERT and AraT5-tweet) generally performed better than those trained on broader multilingual or MSA corpora, especially when the target data (AraDepSu) consists of dialectal tweets.

Limitations and Future Work

  • Limitations:
    • Dialectal Complexity: While MARBERT excels, the paper notes that the confusion between "non-depression" and "suicidal ideation" can still be present, especially with "tricky tweets" that might express general distress without clear suicidal intent. This suggests that capturing subtle nuances and common sense reasoning remains a challenge.
  • Data Bias: The dataset was collected using specific keywords, which might introduce a bias.
  • Future Work:
    • Expanding Dialect Coverage: Further research could focus on models that better handle the vast diversity of Arabic dialects.
  • Improving Nuance Detection: Developing models that can more accurately distinguish between general distress and clinical depression or suicidal ideation is a key area for future improvement.
  • Ethical Considerations: As with any mental health-related AI, future work should also consider the ethical implications of such detection systems.

Summary updated

Abstract

Among mental health diseases, depression is one of the most severe, as it often leads to suicide which is the fourth leading cause of death in the Middle East. In the Middle East, Egypt has the highest percentage of suicidal deaths; due to this, it is important to identify depression and suicidal ideation. In Arabic culture, there is a lack of awareness regarding the importance of diagnosing and living with mental health diseases. However, as noted for the last couple years people all over the world, including Arab citizens, tend to express their feelings openly on social media. Twitter is the most popular platform designed to enable the expression of emotions through short texts, pictures, or videos. This paper aims to predict depression and depression with suicidal ideation. Due to the tendency of people to treat social media as their personal diaries and share their deepest thoughts on social media platforms. Social data contain valuable information that can be used to identify user’s psychological states. We create AraDepSu dataset by scrapping tweets from twitter and manually labelling them. We expand the diversity of user tweets, by adding a neutral label (“neutral”) so the dataset include three classes (“depressed”, “suicidal”, “neutral”). Then we train our AraDepSu dataset on 30+ different transformer models. We find that the best-performing model is MARBERT with accuracy, precision, recall and F1-Score values of 91.20%, 88.74%, 88.50% and 88.75%.
BibTeX
@inproceedings{hassib2022aradepsu,
  author = {Mariam Hassib and Nancy Hossam and Jolie Sameh and Marwan Torki},
  title = {{AraDepSu: Detecting Depression and Suicidal Ideation in Arabic Tweets Using Transformers}},
  booktitle = {WANLP @ EMNLP},
  year = {2022},
  url = {https://openreview.net/forum?id=jVaVE91Pfo},
}
Figure