Robust language-based mental health assessments in time and space through social media

Siddharth Mangalik, Johannes C. Eichstaedt, Salvatore Giorgi, Jihu Mun, Farhan Ahmed, Gilvir Gill, Adithya V. Ganesan, Shashanka Subrahmanya, Nikita Soni, Sean A. P. Clouston, H. Andrew Schwartz

TL;DR

This paper introduces and validates the Language-Based Mental Health Assessment (LBMHA) pipeline, which leverages 1.2 billion tweets to track population-level depression and anxiety at unprecedented spatiotemporal resolutions (down to the county-week level). By demonstrating strong alignment with traditional, expensive national surveys, this method offers a highly scalable, cost-effective framework for real-time public health monitoring and policy evaluation.

What Problem Does This Paper Address?

Traditional public health tracking relies heavily on large-scale, subjective surveys (like those from the CDC or Gallup) to monitor mental health. While valuable, these surveys suffer from severe limitations: they are expensive, infrequent (often annual), and geographically coarse (typically restricted to national or state-level estimates). Furthermore, survey responses can be skewed by social stigma, leading to the under-reporting of mental health struggles.

Because mental health fluctuates rapidly in response to local and global events (e.g., economic shocks, pandemics, or social unrest), public health officials and researchers need a high-resolution, objective, and real-time instrument to identify emerging crises and evaluate the efficacy of local policy interventions.

Methods

The authors developed the Language-Based Mental Health Assessment (LBMHA) pipeline to transform raw social media text into reliable, localized mental health metrics.

Overview of the LBMHA pipeline, from raw tweets to demographic-adjusted county-week scores.

The pipeline consists of four core stages:

  1. Filter and Person Aggregation: Raw tweets are filtered to isolate original, English text (removing retweets and URLs) and grouped by user and county.
  2. Linguistic Pattern Extraction: Word frequencies are computed per user-week and normalized using an Anscombe transformation to stabilize the variance of power-law word distributions.
  3. Depression and Anxiety Scoring: The frequencies are fed into a linear model equipped with a specialized mental health lexicon to generate raw depression (DEP_SCORE) and anxiety (ANX_SCORE) metrics.
  4. Post-Stratification and Aggregation: Because Twitter demographics skew younger, more male, and more highly educated than the general U.S. population, the authors apply post-stratification weights based on county socio-economics to correct for selection bias before aggregating scores to the county-week level.

Novel Adjustments:

  • Target Domain Adaptation: The underlying mental health lexicon was originally trained on Facebook data from the late 2000s. The authors used domain adaptation techniques to filter out words with divergent semantics or frequencies on Twitter (e.g., removing common baby names and words with high distribution shifts), resulting in a refined 5,765-word lexicon.
  • Super-County Binning: To prevent data sparsity in rural areas, county-week observations that fall below a minimum user threshold are aggregated into a state-level "super-county" metric.

Datasets

The study evaluates and validates the pipeline using the following datasets:

  • County Tweet Lexical Bank (CTLB-19-20): A massive, newly updated dataset containing nearly 1 billion processed tweets from 2 million unique geo-located U.S. users across 1,490 counties, spanning all 104 weeks of 2019 and 2020.
  • Gallup COVID-19 Panel Questionnaire: Used as the primary validation benchmark, containing individual survey responses tracking self-reported "sadness" and "worry" across 2,617 counties.
  • County Health Rankings (CHR) 2020: Used for external validity testing against Political, Economic, Social, and Health (PESH) variables (e.g., household income, unemployment, violent crime rates, and election results).

Key Insights and Ideas

  • The Spatiotemporal Sweet Spot: The authors discovered that a county-week resolution strikes the optimal balance between high temporal/spatial granularity and statistical reliability, provided a minimum user threshold is enforced.
  • Macro Events Leave Micro Signatures: Major national disruptions—such as the onset of the COVID-19 pandemic or the social unrest following the murder of George Floyd—cause immediate, sharp spikes in language-estimated depression and anxiety that closely mirror traditional survey trends.
National depression and anxiety scores across 2020, highlighting spikes corresponding to major national events.
  • Exurbs are Struggling: When comparing different American community types, "Exurbs" (communities on the fringe of major metro areas between suburban and rural America) emerged as the most anxious and depressed, highlighting the tool's utility in identifying localized demographic vulnerabilities.

Key Results

  • High Reliability: Split-half reliability testing demonstrated that with a Minimum User Threshold (UT) of 200 users per week, the LBMHA achieved a reliability score $\ge 0.9$. Even at a lower threshold of UT=50, reliability remained above 0.8.
  • Strong Convergent Validity: Multi-level fixed-effects models revealed moderate to large associations between LBMHA scores and Gallup survey metrics down to the county-week level (fixed effects $\beta = 0.25$ to $1.58$; $p < .001$). At the national-week level, the correlation reached $r = 0.39$ for depression/sadness and $r = 0.68$ for anxiety/worry.
  • Sensitivity to Events: Weeks featuring major U.S. events showed an average increase in the mean absolute difference of 23% for depression and 16% for anxiety compared to non-event weeks.
Longitudinal analysis showing the percentage difference in mental health scores during event vs. non-event weeks.

Limitations and Future Work

Despite its robustness, the method has clear limitations:

  • Data Sparsity in Rural Areas: Small counties with low social media activity often fail to meet reliability thresholds, necessitating the "super-county" pooling compromise.
  • Platform Instability: The pipeline is highly dependent on the host platform's data-sharing policies and user base. Recent ownership and policy shifts at Twitter (now X) mean the pipeline requires continuous re-validation for contemporary data.
  • Lexical Evolution: Language changes over time. Future work will need to dynamically update the underlying lexicons to account for evolving slang, emojis, and shifting online communication styles.

Summary updated

Abstract

Compared to physical health, population mental health measurement in the U.S. is very coarse-grained. Currently, in the largest population surveys, such as those carried out by the Centers for Disease Control or Gallup, mental health is only broadly captured through "mentally unhealthy days" or "sadness", and limited to relatively infrequent state or metropolitan estimates. Through the large scale analysis of social media data, robust estimation of population mental health is feasible at much higher resolutions, up to weekly estimates for counties. In the present work, we validate a pipeline that uses a sample of 1.2 billion Tweets from 2 million geo-located users to estimate mental health changes for the two leading mental health conditions, depression and anxiety. We find moderate to large associations between the language-based mental health assessments and survey scores from Gallup for multiple levels of granularity, down to the county-week (fixed effects $β= .25$ to $1.58$; $p<.001$). Language-based assessment allows for the cost-effective and scalable monitoring of population mental health at weekly time scales. Such spatially fine-grained time series are well suited to monitor effects of societal events and policies as well as enable quasi-experimental study designs in population health and other disciplines. Beyond mental health in the U.S., this method generalizes to a broad set of psychological outcomes and allows for community measurement in under-resourced settings where no traditional survey measures - but social media data - are available.
BibTeX
@article{mangalik2024robust,
  author = {Siddharth Mangalik and Johannes C. Eichstaedt and Salvatore Giorgi and Jihu Mun and Farhan Ahmed and Gilvir Gill and Adithya V. Ganesan and Shashanka Subrahmanya and Nikita Soni and Sean A. P. Clouston and H. Andrew Schwartz},
  title = {{Robust language-based mental health assessments in time and space through social media}},
  journal = {Npj Digit. Medicine},
  year = {2024},
  doi = {10.1038/s41746-024-01100-0},
}
Figure