Self-State Evidence Extraction and Well-Being Prediction from Social Media Timelines

Suchandra Chakraborty, Sudeshna Jana, Manjira Sinha, Tirthankar DasguptaNAACL 2025

TL;DR

This paper tackles the challenge of automatically assessing mental well-being from social media by extracting evidence of users' self-states and predicting their well-being scores. They found that while Large Language Models (LLMs) offer contextual understanding, traditional supervised learning methods like RoBERTa and Random Forests provide more reliable and stable predictions for these specific tasks.

What Problem Does This Paper Address?

Mental health assessment using Natural Language Processing (NLP) is moving beyond simple risk classification to understand the dynamic fluctuations of an individual's mental state over time. The CLPsych Shared Task series has been at the forefront of this, progressively introducing more complex challenges. This paper focuses on two key aspects of this evolution: identifying specific pieces of text (spans) that reveal a user's adaptive or maladaptive self-states (e.g., coping mechanisms vs. distress signals) and then using this information to predict an overall well-being score for a given social media post. The ultimate goal is to develop privacy-conscious, resource-efficient methods for psychological assessment that can capture nuanced mental health signals from user-generated content.

Methods

The researchers tackled two main tasks:

  • Task A1: Self-State Evidence Extraction: They framed this as a token classification problem. A RoBERTa model was fine-tuned to label each token in a Reddit post as belonging to an "adaptive" self-state, a "maladaptive" self-state, or "non-evidence." Data preprocessing involved removing posts without evidence and augmenting the dataset with synonym replacement and random word swapping to improve robustness.
  • Task A2: Well-Being Score Prediction: Two distinct approaches were evaluated:
  • LLM-Based Approach (DeepSeek-7B): This involved a retrieval-augmented prompting strategy. For each test post, similar posts from the training data were retrieved based on sentence embeddings. These retrieved examples, along with a detailed prompt explaining the well-being scale (based on the GAF criteria), were fed to the DeepSeek-7B LLM to generate a well-being score.
  • Supervised Learning Approach (Random Forest Regression): Sentence embeddings were generated for each post, and a Random Forest Regressor was trained on these embeddings to predict the well-being score.

Datasets

The study utilized the CLPsych 2025 Shared Task dataset. This dataset comprises user timelines from Reddit, with each post annotated for adaptive and maladaptive self-state evidence spans, as well as an overall well-being score (on a scale of 1-10, similar to the GAF scale). The training set contained 30 user timelines (343 posts), and the test set had 10 user timelines (94 posts).

Key Insights and Ideas

  • Supervised Learning Outperforms LLMs for Numerical Prediction: For the task of predicting a numerical well-being score, traditional supervised learning methods (like Random Forest) proved more reliable and stable than LLM-based approaches.
  • LLMs Excel at Contextual Understanding, But Lack Stability: While LLMs like DeepSeek-7B can capture nuanced mental health signals and provide justifications, they struggle with numerical stability, leading to high variance in predictions and occasional "hallucinations."
  • RoBERTa is Effective for Evidence Extraction: The fine-tuned RoBERTa model demonstrated strong performance in identifying self-state evidence spans, particularly for maladaptive states.
  • Structured Learning is Key for Numerical Tasks: The success of Random Forest is attributed to its utilization of explicit numerical features and supervised training, allowing for precise well-being score prediction.
  • Data Scarcity is a Major Hurdle: The limited size of the annotated dataset presents a significant challenge, impacting the ability of models to capture the full variability of mental health expressions.

Key Results

  • Task A1 (Self-State Evidence Extraction): The RoBERTa-based model achieved a recall of 0.602, placing it second in the shared task. It showed better performance in detecting maladaptive spans (0.681 recall) than adaptive spans (0.522 recall).
  • Task A2 (Well-Being Score Prediction):
    • The Random Forest Regression model achieved the second-lowest Mean Squared Error (MSE) of 2.994, outperforming most baselines and the DeepSeek-7B model.
  • The DeepSeek-7B LLM model struggled with numerical stability, resulting in a higher MSE of 6.610, particularly in severe distress cases.

Limitations and Future Work

Limitations:

  • Data Scarcity: The dataset size limits the models' ability to capture the full spectrum of mental health expressions.
  • LLM Hallucinations and Instability: LLMs, while good at contextual reasoning, exhibited unreliable numerical predictions and occasional hallucinations.
  • Span vs. Sentence Classification: The RoBERTa model was fine-tuned for sentence classification, leading to difficulties in accurately identifying specific evidence spans within sentences, especially when mixed sentiments were present.
  • Ambiguity in Language: The model struggled with ambiguous language and negations, leading to misclassifications.

Future Work:

  • Span-Level Annotation: Moving towards span-level annotation for Task A1 to better distinguish cues within sentences.
  • LLM Data Augmentation: Exploring LLM-based data augmentation for Task A1 to generate more diverse training instances.
  • Incorporating Stylistic Features: Including sentiment shifts, discourse markers, and writing patterns to enhance contextual understanding.
  • Hybrid Models: Developing hybrid models that combine LLM contextual reasoning with the stability of regression-based approaches.
  • Temporal Trend Analysis: Extending models to capture temporal trends in user well-being for more robust longitudinal assessment.
  • Improving LLM Efficiency: Investigating ways to enhance LLM performance through domain-specific fine-tuning, improved retrieval mechanisms, and constrained numerical outputs.

Summary updated

BibTeX
@inproceedings{chakraborty2025self,
  author = {Suchandra Chakraborty and Sudeshna Jana and Manjira Sinha and Tirthankar Dasgupta},
  title = {{Self-State Evidence Extraction and Well-Being Prediction from Social Media Timelines}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2025},
}
Figure