Exploring transformers and time lag features for predicting changes in mood over time

John Culnan, Damian Romero Diaz, Steven Bethard2022 · DOI

TL;DR

Researchers from the University of Arizona investigated whether adding explicit temporal features—like the time elapsed between posts—improves Transformer-based models' ability to predict mood changes in Reddit users. Their findings suggest that while a standard RoBERTa model is remarkably robust on its own, adding "time lag" features provides a specialized boost in detecting sudden switches in emotional state.

What Problem Does This Paper Address?

Mental health professionals and online community moderators have a growing interest in "digital phenotyping"—using social media activity to identify individuals at risk. However, mental health isn't a static label; it’s a moving target. The CLPsych 2022 shared task challenged researchers to move beyond static classification and instead predict moments of change in a user's timeline.

The core challenge is longitudinal: does the model need to "remember" the sequence of posts and the specific time gaps between them to understand if a user is escalating (getting worse/better) or switching (changing mood entirely)? The authors specifically address whether complex sequential architectures (like LSTMs) or explicit time-gap features actually outperform a standard, "timeline-agnostic" Transformer.

Methods

The authors compared several architectures built on top of RoBERTa-base:

  1. Timeline-Agnostic Models: These treat posts as independent data points.
  2. RoBERTa (Baseline): Uses the standard [CLS] token embedding.
  3. RoBERTa-lin-lag: A novel approach where the model takes the post embedding and concatenates a "time lag" feature—the number of seconds since the user's previous post—before passing it through a linear layer.
  4. Timeline-Preserving Models: These look at a sliding window of three posts (the current post plus the two previous ones).
  5. RoBERTa-pre2-lin: Concatenates the embeddings of all three posts into one large vector.
  6. RoBERTa-pre2-lstm: Feeds the sequence of three post embeddings into a Long Short-Term Memory (LSTM) network to capture sequential dependencies.

Datasets

The study utilized a curated collection of Reddit data provided for the CLPsych 2022 shared task, which includes:

  • UMD Reddit Suicidality Dataset (v2)
  • Reddit-New: A dataset of posts from mental-health-related subreddits.
  • eRisk Dataset

The data consists of 4,775 posts labeled with three classes: IS (Switch in mood), IE (Escalation in mood), and O (No change/Baseline). The dataset is heavily imbalanced, with the "No change" class (O) making up roughly 80% of the training data.

Key Insights and Ideas

  • The "Simplicity" Paradox: Surprisingly, adding complex sequential layers like LSTMs actually degraded performance. The LSTM model struggled so much with the data imbalance that it eventually defaulted to only predicting the majority "No change" class.
  • Time Lag as a Specialized Signal: Explicitly encoding the time between posts (the "lag") didn't overhaul the model's overall accuracy, but it significantly improved the Precision for the "Switch in mood" (IS) category. This suggests that the duration of silence between posts is a meaningful feature for identifying sudden shifts in mental state.
  • Transformer "Intuition": The baseline RoBERTa model performed remarkably well without any knowledge of the timeline. This implies that the linguistic content of a single post often contains enough "state" information for the model to infer a change without needing to see the previous posts.
  • Contextual Noise: Qualitative analysis revealed that models often hallucinate mood changes when a user discusses past trauma or expresses empathy for someone else's depression, highlighting a need for better temporal grounding in NLP.

Key Results

The researchers evaluated models using Macro F1 scores and coverage-based metrics.

  • Top Performers: The RoBERTa baseline and RoBERTa-lin-lag were the clear winners. On the test set, they achieved nearly identical Macro F1 scores (.510 vs .507).
  • The Trade-off: The baseline RoBERTa was better at identifying Escalations (IE), while the model with the time-lag feature (RoBERTa-lin-lag) was significantly better at identifying Switches (IS), achieving a Precision of .267 compared to the baseline's .142.
  • Baseline Comparison: Both Transformer models comfortably outperformed traditional baselines like Logistic Regression with TF-IDF features.

Limitations and Future Work

The authors noted several hurdles:

  • Memory Constraints: Due to hardware limits, the "timeline-preserving" models could only look back at a window of two previous posts. A larger context window might reveal more complex longitudinal patterns.
  • The "Empty Post" Problem: The models struggled with posts that had a title but no body text. In these cases, the models defaulted to "No change," even when the title itself was highly indicative of a mood shift.
  • Linguistic Nuance: Future work needs to focus on distinguishing between a user's current emotional state and their descriptions of past events or external situations, which currently trigger false positives.

Summary updated

Abstract

This paper presents transformer-based models created for the CLPsych 2022 shared task. Using posts from Reddit users over a period of time, we aim to predict changes in mood from post to post. We test models that preserve timeline information through explicit ordering of posts as well as those that do not order posts but preserve features on the length of time between a user's posts. We find that a model with temporal information may provide slight benefits over the same model without such information, although a RoBERTa transformer model provides enough information to make similar predictions without custom-encoded time information.
BibTeX
@misc{culnan2022exploring,
  author = {John Culnan and Damian Romero Diaz and Steven Bethard},
  title = {{Exploring transformers and time lag features for predicting changes in mood over time}},
  year = {2022},
  doi = {10.18653/v1/2022.clpsych-1.21},
}
Figure