Measuring Mental Health Variables in Computational Research: Toward Validated, Dimensional, and Transdiagnostic Approaches

Chen Shani, Elizabeth C. Stade

Introduction

In the rapidly evolving field of computational mental health, the integration of machine learning (ML) and natural language processing (NLP) holds the promise of revolutionizing diagnostics and treatment. However, a significant "garbage in, garbage out" problem persists. While computational techniques have become increasingly sophisticated, the quality of the labels used to train these models often lags behind. This paper addresses the widening gap between computer science methodologies and clinical psychology, proposing a framework for researchers to adopt more robust measurement practices.

The Problem: The Proxy Trap and Construct Validity

Much of the current research in computational mental health relies on "proxy signals" for ground truth. These include convenience measures such as membership in specific subreddits (e.g., r/Depression) or self-disclosures in tweets. While accessible, these labels often lack construct validity—the degree to which a test actually measures what it claims to measure.

For instance, a user in a depression-related forum might be a caregiver, a researcher, or an individual experiencing transient sadness rather than clinical depression. When models are trained on these "invalid" labels, their predictions become fundamentally flawed, limiting their external validity and making them unsuitable for clinical application.

A Framework for Improved Measurement

To bridge this gap, the authors synthesize decades of clinical science into a conceptual framework based on three core pillars: measurement validity, dimensionality, and transdiagnostic approaches.

1. Prioritizing Measurement Validity

Researchers are encouraged to move away from ad-hoc proxy labels and toward "gold standard" assessments. This includes:

  • Clinician-administered interviews: Such as the Structured Clinical Interview for DSM-5 (SCID-5).
  • Psychometrically validated self-reports: Such as the Patient Health Questionnaire-9 (PHQ-9).

Even small shifts, such as using a validated single-item measure instead of a home-grown one, can significantly enhance a model's validity.

2. Embracing Dimensionality

Mental health symptoms exist on a continuous spectrum rather than as binary states. However, most computational research treats mental health as a categorical variable (e.g., "depressed" vs. "healthy"). This practice of dichotomization has several negative consequences:

  • Loss of Statistical Power: Reducing a continuous range to a binary label sacrifices nuance and reduces the reliability of the model.
  • Type II Errors: Categorical models are more likely to miss real effects.
  • Boundary Case Neglect: Binary cutoffs ignore "subthreshold" cases—individuals who experience significant distress but do not meet the arbitrary threshold for a formal diagnosis.

Dimensional models that predict severity scores (e.g., a scale of 0–10) capture this meaningful variation and align more closely with the underlying structure of psychological conditions.

3. Adopting Transdiagnostic Frameworks

Traditional diagnostic categories often overlap. The authors advocate for looking at processes that cut across different disorders—such as "avoidance" or "negative affect." They point toward established frameworks like the Research Domain Criteria (RDoC) and the Hierarchical Taxonomy of Psychopathology (HiTOP) as guides for identifying these cross-cutting features.

The Matrix of Measures

The paper introduces a "Matrix of Measures" to help researchers evaluate their data collection strategies. Using social anxiety as a case study, the matrix categorizes strategies across three axes:

  • Validated vs. Unvalidated: Comparing clinician-rated tools against social media disclosures.
  • Categorical vs. Dimensional: Comparing binary status against continuous severity scores.
  • Syndrome-specific vs. Transdiagnostic: Comparing anxiety-specific measures against general fear subfactors.

Challenges and Future Directions

Transitioning to these high-quality measurement practices is not without hurdles. Clinician-assessed data is significantly more expensive and time-consuming to collect than scraping social media. Furthermore, clinical data involves Protected Health Information (PHI), raising complex privacy and ethical concerns regarding data sharing and aggregation.

To move forward, the authors call for the development of large-scale, shared datasets derived from clinical practice-research networks. These datasets should ideally include natural language data (such as interview transcripts) paired with validated severity scores. Future empirical studies should also focus on quantifying the "cost" of bad data by directly comparing the performance of models trained on proxy signals versus those trained on gold-standard clinical labels.

By aligning computational goals with clinical realities, the field can move toward building tools that are not only technically impressive but also clinically useful and scientifically sound.

Summary updated

BibTeX
@inproceedings{shani2025measuring,
  author = {Chen Shani and Elizabeth C. Stade},
  title = {{Measuring Mental Health Variables in Computational Research: Toward Validated, Dimensional, and Transdiagnostic Approaches}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2025},
}
Figure