Can We Assess Mental Health Through Social Media and Smart Devices? Addressing Bias in Methodology and Evaluation

Adam Tsakalidis, Maria Liakata, Theo Damoulas, Alexandra I. Cristea

TL;DR

This paper critically examines how we evaluate mental health prediction models using social media and smartphone data. It reveals that current state-of-the-art methods, when tested under realistic conditions, perform poorly and barely outperform simple baselines, raising serious questions about their real-world applicability and the utility of the features used.

What Problem Does This Paper Address?

The promise of using digital footprints from smartphones and social media to monitor mental well-being is immense, potentially revolutionizing how we assess and manage mental health. Existing research often reports highly accurate predictions of mental health states (like depression or happiness) using these data sources. However, this paper questions whether these impressive results translate to real-world scenarios. The core problem addressed is the bias in evaluation methodologies commonly used in this field. These biases can lead to overly optimistic performance estimates that don't reflect how these models would actually perform when deployed to assess individuals not seen during training, or when trying to predict future states. The authors aim to expose these flaws and propose a more pragmatic evaluation framework.

Methods

The paper's primary contribution lies in its rigorous re-evaluation of existing state-of-the-art approaches using different evaluation settings. They focus on three key problems stemming from unrealistic evaluation:

  • P1: Training on Past Values of the Target Variable: This involves using previous mental health scores to predict future ones. The authors follow the methodology of LiKamWa et al. [13] to investigate this, using Linear Regression with Sequential Feature Selection.
  • P2: Inferring Test Labels: This addresses the issue of temporal leakage, where features used for training are derived from time periods that overlap with the test set's time period, leading to inflated performance. They replicate the approach of Canzian and Musolesi [3], using SVM classifiers with varying time windows for feature extraction.
  • P3: Predicting Users Instead of Mood Scores: This highlights the danger of models learning to identify individual users rather than their mental state, especially when using user-specific features (like text from social media). They replicate experiments from Tsakalidis et al. [26] and Jaques et al. [9], employing SVM (SVRRBF and standard SVM) and comparing performance across different validation settings (MIXED, LOIOCV, LOUOCV), with and without user-specific feature normalization.

Datasets

The study utilizes two publicly available datasets:

  • Dataset 1: Derived from Tsakalidis et al. [26], this dataset includes longitudinal textual data (social media posts, private messages) and mobile phone usage data from 30 subjects. The ground truth for mental health includes positive, negative, and well-being mood scores.
  • Dataset 2: The StudentLife dataset [28], comprising data from 48 students over 10 weeks. This includes smartphone activity logs (activity, conversation timestamps, audio mode, phone status) and self-reported stress levels as the target variable.

Key Insights and Ideas

  • "Real-world" evaluation is crucial: The paper strongly argues that standard evaluation methods (like MIXED cross-validation) often create a false sense of high performance that doesn't hold up in pragmatic, unbiased settings.
  • User bias is a major confounder: When models are trained and tested on the same users (even if split into train/test sets), they can learn to identify users rather than their mental state, especially with personalized features like text. This leads to inflated accuracy.
  • Temporal leakage inflates performance: Using features derived from time periods that overlap with the target prediction period (even if the target itself isn't directly used as a feature) can lead to over-optimistic results, particularly in LOIOCV settings.
  • Naïve baselines are surprisingly competitive: In unbiased settings (LOUOCV), state-of-the-art models often perform only marginally better than simple baselines like predicting the user's average score or the last recorded score.
  • Feature contribution is questionable: The poor performance in unbiased settings raises doubts about the actual contribution of many sophisticated features derived from social media and sensors for mental health assessment.

Key Results

  • Performance drops dramatically in unbiased settings: Across all three problem statements (P1, P2, P3), when evaluated using pragmatic, unbiased settings (primarily LOUOCV), the performance of the models drops significantly.
  • Barely outperforming baselines: In many cases, the models could barely outperform simple baselines like predicting the average mood score for a user (AVG) or the last known score (LAST). For instance, in P1, the AVG baseline often performed better than or comparably to the tested models in LOUOCV.
  • User bias demonstrated: In P3, when models were trained to identify users (using user ID as a feature), they achieved high R² values (e.g., 0.64 for "wellbeing"), demonstrating that the model was primarily predicting the user, not their mental state. Removing this user bias led to negative R² values in LOIOCV.
  • Temporal bias evident: In P2, the paper shows that replacing input features with random noise still yielded comparable results to using actual features in certain settings, highlighting the temporal bias that inflates performance.

Limitations and Future Work

  • Limitations: The paper acknowledges that the magnitude of the observed effects might vary across different datasets and feature types. The primary limitation is the difficulty in achieving high performance in unbiased, real-world settings with current methods.
  • Future Work: The authors propose establishing appropriate evaluation settings to avoid false alarms and build deployable systems. They advocate for:
  • LOUOCV: Training on a sample of users and testing on completely new users, ensuring independence.
  • LOIOCV: Ensuring models do not violate the i.i.d. hypothesis and consistently outperform naïve baselines.
  • Transfer Learning: Exploring transfer learning approaches for LOUOCV settings, especially for latent feature representations, to overcome the lack of data for individual users.
  • Larger Datasets: Utilizing larger datasets to better assess generalization capabilities.
  • Better Feature Engineering: Investigating latent feature representations that might capture underlying behavioral patterns across users.

Summary updated

Abstract

Predicting mental health from smartphone and social media data on a longitudinal basis has recently attracted great interest, with very promising results being reported across many studies [3, 9, 13, 26]. Such approaches have the potential to revolutionise mental health assessment, if their development and evaluation follows a real world deployment setting. In this work we take a closer look at state-of-the-art approaches, using different mental health datasets and indicators, different feature sources and multiple simulations, in order to assess their ability to generalise. We demonstrate that under a pragmatic evaluation framework, none of the approaches deliver or even approach the reported performances. In fact, we show that current state-of-the-art approaches can barely outperform the most naïve baselines in the real-world setting, posing serious questions not only about their deployment ability, but also about the contribution of the derived features for the mental health assessment task and how to make better use of such data in the future.
BibTeX
@inproceedings{tsakalidis2018can,
  author = {Adam Tsakalidis and Maria Liakata and Theo Damoulas and Alexandra I. Cristea},
  title = {{Can We Assess Mental Health Through Social Media and Smart Devices? Addressing Bias in Methodology and Evaluation}},
  booktitle = {ECML/PKDD},
  year = {2018},
  url = {https://openreview.net/forum?id=rGzWqgx8qCF},
}
Figure