Enhancing Suicide Risk Detection on Social Media through Semi-Supervised Deep Label Smoothing

Matthew R. Squires, Xiaohui Tao, Soman Elangovan, U. Rajendra Acharya, Raj Gururajan, Haoran Xie, Xujuan ZhouarXiv 2024 · DOI · arXiv

Abstract

Detecting suicide risk on social media is a critical task complicated by the inherent subjectivity of mental health assessments. Traditional machine learning approaches often rely on "hard labels" that fail to account for annotator disagreement and data ambiguity. This paper introduces a semi-supervised framework, Deep Bayesian Label Smoothing, which utilizes Monte Carlo Dropout to estimate uncertainty and generate "soft" labels. Experimental results on the Reddit C-SSRS dataset demonstrate that this approach significantly improves classification performance, raising accuracy from 43.12% to 52.33% and dramatically enhancing balanced accuracy.

Introduction

The detection of suicide risk from social media text is a high-stakes application of AI. However, unlike many medical diagnostic tasks, mental health conditions lack objective biomarkers, leading to significant subjectivity in expert annotations. Annotator agreement for suicide risk levels can be as low as 60%.

Standard machine learning models typically employ one-hot encoded "hard labels," forcing a binary certainty (e.g., 100% "Ideation") that ignores "aleatoric uncertainty"—the noise and ambiguity inherent in the data. This overconfidence can lead to poor generalization. This work proposes that embracing this uncertainty through label smoothing can better capture the "fuzzy" reality of clinical diagnosis and improve model robustness.

Methods: Deep Bayesian Label Smoothing

The researchers developed a three-stage semi-supervised framework to integrate uncertainty into the learning process.

  1. Initial Training: A Convolutional Neural Network (CNN) is first trained using the original expert-provided "hard labels."
  2. Uncertainty Estimation: To quantify the model's confidence, the framework employs Monte Carlo (MC) Dropout. By maintaining active dropout during the inference phase and performing 100 stochastic passes for each post, the system generates a distribution of predictions rather than a single point estimate.
  3. Label Smoothing and Retraining: The mean of these 100 predictions forms a "soft label"—a probability distribution across all risk categories (e.g., 70% Ideation, 25% Behavior, 5% Indicator). The model is then retrained using these data-driven soft labels, which reflect the specific ambiguity of each individual post.
Architecture of the Deep Bayesian Label Smoothing framework.

Unlike standard uniform label smoothing, which redistributes probability mass equally, this Bayesian approach is non-uniform and context-specific, allowing the model to learn from the inherent overlap between clinical categories.

Datasets

The study utilizes the Reddit C-SSRS dataset, which consists of posts from 500 users labeled according to the Columbia Suicide Severity Rating Scale (C-SSRS). The dataset includes five classes: Suicidal Ideation (ID), Suicidal Behavior (SB), Actual Attempt (AT), Suicide Indicator (IN), and Supportive (SU). The dataset is notably imbalanced, with "Ideation" being the most frequent class, posing a significant challenge for standard classifiers.

Results

The proposed method established a new state-of-the-art for the Reddit C-SSRS dataset. A key finding was that the baseline model was "over-confident" in the most frequent class (Ideation), predicting it 92% of the time. By introducing uncertainty, the new model became more balanced.

Metric Baseline Deep Bayesian Label Smoothing
Accuracy 43.12% 52.33%
Weighted Balanced Accuracy 25.67% 49.23%
Macro Average Precision - 47.21%

The dramatic improvement in weighted balanced accuracy indicates that the model became significantly more effective at identifying minority risk levels rather than simply defaulting to the most common label.

Discussion and Limitations

While the framework shows clear benefits, several challenges remain. The model still struggled with the "Actual Attempt" (AT) class, yielding zero true positives due to extreme data scarcity. Future work may involve replacing standard word embeddings with transformer models like BERT to better capture contextual nuance and addressing the computational cost of MC Dropout. By treating subjectivity as a feature rather than noise, this research provides a more nuanced path forward for AI-driven mental health monitoring.

Summary updated

Abstract

Suicide is a prominent issue in society. Unfortunately, many people at risk for suicide do not receive the support required. Barriers to people receiving support include social stigma and lack of access to mental health care. With the popularity of social media, people have turned to online forums, such as Reddit to express their feelings and seek support. This provides the opportunity to support people with the aid of artificial intelligence. Social media posts can be classified, using text classification, to help connect people with professional help. However, these systems fail to account for the inherent uncertainty in classifying mental health conditions. Unlike other areas of healthcare, mental health conditions have no objective measurements of disease often relying on expert opinion. Thus when formulating deep learning problems involving mental health, using hard, binary labels does not accurately represent the true nature of the data. In these settings, where human experts may disagree, fuzzy or soft labels may be more appropriate. The current work introduces a novel label smoothing method which we use to capture any uncertainty within the data. We test our approach on a five-label multi-class classification problem. We show, our semi-supervised deep label smoothing method improves classification accuracy above the existing state of the art. Where existing research reports an accuracy of 43\% on the Reddit C-SSRS dataset, using empirical experiments to evaluate our novel label smoothing method, we improve upon this existing benchmark to 52\%. These improvements in model performance have the potential to better support those experiencing mental distress. Future work should explore the use of probabilistic methods in both natural language processing and quantifying contributions of both epistemic and aleatoric uncertainty in noisy datasets.
BibTeX
@misc{squires2024enhancing,
  author = {Matthew R. Squires and Xiaohui Tao and Soman Elangovan and U. Rajendra Acharya and Raj Gururajan and Haoran Xie and Xujuan Zhou},
  title = {{Enhancing Suicide Risk Detection on Social Media through Semi-Supervised Deep Label Smoothing}},
  year = {2024},
  eprint = {2405.05795},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2405.05795},
}
Figure