Prompt Engineering for Capturing Dynamic Mental Health Self States from Social Media Posts

Callum Chan, Sunveer Khunkhun, Diana Inkpen, Juan Antonio Lossio-VenturaNAACL 2025

TL;DR

This paper explores how to use prompt engineering with large language models (LLMs) to understand and summarize users' dynamic mental health states from social media. The authors achieved top rankings in a shared task by demonstrating that carefully crafted prompts can effectively guide LLMs to identify emotional nuances and generate coherent summaries of mental well-being over time.

What Problem Does This Paper Address?

The paper addresses the challenge of accurately capturing and summarizing the complex and evolving nature of mental health states as expressed in social media. While LLMs show promise in analyzing text for mental health indicators, understanding the dynamics of these states—how they change over time and the underlying reasons—remains a significant hurdle. This is crucial for developing more personalized and effective mental health support. The authors specifically tackle the CLPsych 2025 shared task, which involves identifying textual evidence of adaptive and maladaptive self-states, scoring well-being, and generating both post-level and timeline-level summaries of these states.

Methods

The core of the authors' approach is prompt engineering applied to the Llama-3.3-70B-Instruct large language model. They experimented with three main prompting strategies:

  • Zero-shot prompting: Providing only the task instructions without examples.
  • Structured contextual one-shot prompting: Including task requirements, definitions, guidelines, and a single annotated example.
  • Structured contextual few-shot prompting: Similar to one-shot, but with multiple annotated examples (seven for some tasks, three for others).

For the well-being scoring task (Task A.2), they also explored an additional approach involving regression models. This involved generating embeddings from various transformer-based LLMs (including BERT, RoBERTa, MentalBERT, MentalRoBERTa, SBERT, and Llama-3.3-70B-Instruct) and then training linear (LR) and multinomial logistic regression (MLR) classifiers on these embeddings.

Datasets

The study utilized data from the CLPsych 2025 shared task. This dataset consists of:

  • Training data: Thirty timelines of posts, totaling 343 posts, fully annotated with gold evidence for self-states, well-being scores, and post/timeline summaries.
  • Test data: Ten additional timelines, totaling 94 posts, used for evaluation.

The paper does not mention introducing any newly created benchmarks.

Key Insights and Ideas

  • Prompting is Powerful: The study strongly suggests that prompt engineering is a highly effective method for guiding LLMs in nuanced mental health analysis tasks, often outperforming simpler approaches.
  • One-Shot Learning Shines for Extraction: For tasks requiring precise identification of evidence (like Task A.1), a single, well-chosen example in a structured prompt (one-shot) was more effective than multiple examples (few-shot) or no examples (zero-shot). This suggests that too many examples can introduce confusion.
  • Few-Shot Learning Aids Nuance: For tasks requiring more complex reasoning or summarization (like Task B and C), few-shot prompting, with multiple examples, demonstrated superior performance, especially for capturing temporal dynamics in timeline summaries.
  • Regression Models Excel for Scoring: While prompt engineering was effective, regression models trained on LLM embeddings achieved the best results for numerical well-being scoring (Task A.2), indicating that direct statistical pattern learning from embeddings can be more robust for quantitative tasks than prompting alone.
  • LLMs Understand Mental Health Concepts: The results demonstrate that LLMs, when properly prompted, can effectively identify and interpret subtle indicators of adaptive and maladaptive self-states within social media text.

Key Results

The uOttawa team achieved impressive rankings in the CLPsych 2025 shared task:

  • Task A.1 (Self-State Identification): Placed first with their one-shot prompting strategy, achieving a recall of 0.637.
  • Task A.2 (Well-Being Score): Placed sixth with their few-shot prompting strategy (MSE of 2.62). However, their additional regression experiments (S4, S5, S6) achieved even lower MSEs (down to 1.673), outperforming all other teams.
  • Task B (Post-Level Summaries): Placed fifth with their one-shot prompting strategy, achieving a mean consistency of 0.860.
  • Task C (Timeline-Level Summaries): Placed second with their few-shot prompting strategy, achieving a mean consistency of 0.943.

Limitations and Future Work

Limitations: * Data Specificity: The study's findings are based on social media data and are limited to English-language posts. * LLM Choice: Only the Llama LLM was tested, with a small version for baseline and a larger one for the main experiments. * Prompt Bias: The authors acknowledge that their prior experience and the way they defined terms in prompts might have introduced biases. * Temporal Dynamics: The current approach scored posts sequentially but did not condition predictions on previous post predictions, limiting the detection of extended periods of specific self-states. * Ablation Studies: Due to time constraints, detailed ablation studies on prompt components and hyperparameter tuning (temperature, top_p) were not performed.

Future Work: * Prompt Optimization: Further explore few-shot learning by varying the number of examples and their selection. * LLM Exploration: Experiment with different LLM types and sizes, and consider pre-training or fine-tuning LLMs on mental health-specific data. * Sequential Modeling for Scoring: For Task A.2, develop models that incorporate sequences of decisions and leverage features from previous posts. * Extended Temporal Analysis: Develop methods to detect extended periods of adaptive or maladaptive self-states. * Bias Mitigation: Investigate and mitigate potential biases introduced by prompt engineering.

Summary updated

BibTeX
@inproceedings{chan2025prompt,
  author = {Callum Chan and Sunveer Khunkhun and Diana Inkpen and Juan Antonio Lossio-Ventura},
  title = {{Prompt Engineering for Capturing Dynamic Mental Health Self States from Social Media Posts}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2025},
}
Figure