Abstract
Detecting suicide risk on social media is a critical task complicated by the inherent subjectivity of mental health assessments. Traditional machine learning approaches often rely on "hard labels" that fail to account for annotator disagreement and data ambiguity. This paper introduces a semi-supervised framework, Deep Bayesian Label Smoothing, which utilizes Monte Carlo Dropout to estimate uncertainty and generate "soft" labels. Experimental results on the Reddit C-SSRS dataset demonstrate that this approach significantly improves classification performance, raising accuracy from 43.12% to 52.33% and dramatically enhancing balanced accuracy.
Introduction
The detection of suicide risk from social media text is a high-stakes application of AI. However, unlike many medical diagnostic tasks, mental health conditions lack objective biomarkers, leading to significant subjectivity in expert annotations. Annotator agreement for suicide risk levels can be as low as 60%.
Standard machine learning models typically employ one-hot encoded "hard labels," forcing a binary certainty (e.g., 100% "Ideation") that ignores "aleatoric uncertainty"—the noise and ambiguity inherent in the data. This overconfidence can lead to poor generalization. This work proposes that embracing this uncertainty through label smoothing can better capture the "fuzzy" reality of clinical diagnosis and improve model robustness.
Methods: Deep Bayesian Label Smoothing
The researchers developed a three-stage semi-supervised framework to integrate uncertainty into the learning process.
- Initial Training: A Convolutional Neural Network (CNN) is first trained using the original expert-provided "hard labels."
- Uncertainty Estimation: To quantify the model's confidence, the framework employs Monte Carlo (MC) Dropout. By maintaining active dropout during the inference phase and performing 100 stochastic passes for each post, the system generates a distribution of predictions rather than a single point estimate.
- Label Smoothing and Retraining: The mean of these 100 predictions forms a "soft label"—a probability distribution across all risk categories (e.g., 70% Ideation, 25% Behavior, 5% Indicator). The model is then retrained using these data-driven soft labels, which reflect the specific ambiguity of each individual post.
Unlike standard uniform label smoothing, which redistributes probability mass equally, this Bayesian approach is non-uniform and context-specific, allowing the model to learn from the inherent overlap between clinical categories.
Datasets
The study utilizes the Reddit C-SSRS dataset, which consists of posts from 500 users labeled according to the Columbia Suicide Severity Rating Scale (C-SSRS). The dataset includes five classes: Suicidal Ideation (ID), Suicidal Behavior (SB), Actual Attempt (AT), Suicide Indicator (IN), and Supportive (SU). The dataset is notably imbalanced, with "Ideation" being the most frequent class, posing a significant challenge for standard classifiers.
Results
The proposed method established a new state-of-the-art for the Reddit C-SSRS dataset. A key finding was that the baseline model was "over-confident" in the most frequent class (Ideation), predicting it 92% of the time. By introducing uncertainty, the new model became more balanced.
| Metric | Baseline | Deep Bayesian Label Smoothing |
|---|---|---|
| Accuracy | 43.12% | 52.33% |
| Weighted Balanced Accuracy | 25.67% | 49.23% |
| Macro Average Precision | - | 47.21% |
The dramatic improvement in weighted balanced accuracy indicates that the model became significantly more effective at identifying minority risk levels rather than simply defaulting to the most common label.
Discussion and Limitations
While the framework shows clear benefits, several challenges remain. The model still struggled with the "Actual Attempt" (AT) class, yielding zero true positives due to extreme data scarcity. Future work may involve replacing standard word embeddings with transformer models like BERT to better capture contextual nuance and addressing the computational cost of MC Dropout. By treating subjectivity as a feature rather than noise, this research provides a more nuanced path forward for AI-driven mental health monitoring.
Summary updated