Machine learning identifies a COVID-19-specific phenotype in university students using a mental health app

Artur Shvetcov, Alexis E. Whitton, Suranga Kasturi, Wu-Yi Zheng, Joanne R. Beames, Omar Ibrahim, Jin Han, Leonard Hoon, Kon Mouzakis, Sunil Gupta, Svetha Venkatesh, Helen Christensen, Jill M. NewbymedRxiv 2022 · DOI

Abstract

Advances in smartphone technology have allowed people to access mental healthcare via digital apps from wherever and whenever they choose. University students experience a high burden of mental health concerns. Although these apps improve mental health symptoms, user engagement has remained low. Studies have shown that users can be subgrouped based on unique characteristics that just-in-time adaptive interventions (JITAIs) can use to improve engagement. To date, however, no studies have examined the effect of the COVID-19 pandemic on these subgroups. Here, we use machine learning to examine user subgroup characteristics across three COVID-19-specific timepoints: during lockdown, immediately following lockdown, and three months after lockdown ended. We demonstrate that there are three unique subgroups of university students who access mental health apps. Two of these, with either higher or lower mental well-being, were defined by characteristics that were stable across COVID-19 timepoints. The third, situational well-being, had characteristics that were timepoint-dependent, suggesting that they are highly influenced by traumatic stressors and stressful situations. This subgroup also showed feelings and behaviours consistent with burnout. Overall, our findings clearly suggest that user subgroups are unique: they have different characteristics and therefore likely have different mental healthcare goals. Our findings also highlight the importance of including questions and additional interventions targeting traumatic stress(ors), reason(s) for use, and burnout in JITAI-style mental health apps to improve engagement.

Introduction

Digital mental health applications offer a scalable solution for university students, a demographic facing a disproportionate burden of anxiety and depression. However, these apps suffer from a major "engagement gap"—users often stop using them before seeing benefits. To improve engagement, developers are turning to Just-In-Time Adaptive Interventions (JITAIs), which tailor support to a user’s specific state.

A critical challenge for JITAIs is understanding how major external stressors—like a global pandemic—reshape user subgroups. If a user’s phenotype changes because of a lockdown, a static intervention model will fail. This study investigates whether user phenotypes are stable traits or fluid states influenced by the environment, specifically focusing on the impact of the COVID-19 pandemic.

Methodology

The researchers employed a multi-stage machine learning and statistical pipeline to analyze user data across three pandemic phases. The data was sourced from the "Vibe Up" smartphone app trial involving Australian university students, partitioned into three datasets: Lockdown ($n=308$), Post-lockdown ($n=84$), and Normal ($n=81$, collected three months after restrictions ended).

Machine Learning and Statistical Pipeline: This flowchart illustrates the progression from data pre-processing to unsupervised and supervised machine learning, concluding with inferential statistics to validate user clusters.

As shown in the pipeline above, the analysis involved:

  1. Unsupervised Learning: K-means clustering and Principal Component Analysis (PCA) were used to identify natural groupings and feature correlations within the student population.
  2. Supervised Learning: Shrinkage Discriminant Analysis (SDA) and decision trees were used to determine if these groups were predictable. Models were cross-tested by training on one pandemic timepoint and testing on another to assess stability.
  3. Feature Selection: Recursive Feature Elimination (RFE) ranked the importance of 46 psychological and demographic variables, such as distress (K10), stress (PSS), and well-being (WBS).

Results: Stable Traits vs. Fluid States

The analysis identified three distinct subgroups of university students. Two of these groups—Cluster 1 (High Well-being) and Cluster 2 (Low Well-being)—remained remarkably stable. SDA models trained on lockdown data predicted members of these groups in the "normal" dataset with high accuracy (F1-scores and precision metrics around 0.90), suggesting these phenotypes are driven more by internal traits than external circumstances.

In contrast, Cluster 3 (Situational Well-being) acted as a "chameleon" subgroup. During lockdowns, this group’s profile mirrored the low well-being group, but in "normal" times, they resembled the high well-being group. When attempting to predict Cluster 3 membership using models trained on different timepoints, performance often dropped to chance levels, highlighting how significantly the environment altered this group's profile.

The Burnout Phenotype

In the period immediately following lockdown, Cluster 3 displayed unique behaviors, including increased alcohol/tobacco use and a reliance on family support despite high distress. The researchers identified this as a "burnout" phase, where the effects of a prolonged stressor persist even after the stressor is removed. This suggests a specific phenotype of students who are generally resilient but "break" under high environmental stress.

Additionally, the study found a significant portion of app users already possessed high well-being. This indicates a market for mental health apps focused on maintenance and prevention rather than just crisis intervention.

Conclusion and Future Directions

While the study is limited by self-selection bias and the use of self-report surveys rather than clinical assessments, it provides a roadmap for more effective digital interventions. The findings suggest that JITAI-style apps should move beyond generic depression and anxiety modules. Instead, they should include explicit questions about the user's reason for use (crisis vs. maintenance) and incorporate specific interventions for burnout and traumatic stress, as these appear to be distinct states that traditional models might miss.

Summary updated

Abstract

Abstract Advances in smartphone technology have allowed people to access mental healthcare via digital apps from wherever and whenever they choose. University students experience a high burden of mental health concerns. Although these apps improve mental health symptoms, user engagement has remained low. Studies have shown that users can be subgrouped based on unique characteristics that just-in-time adaptive interventions (JITAIs) can use to improve engagement. To date, however, no studies have examined the effect of the COVID-19 pandemic on these subgroups. Here, we use machine learning to examine user subgroup characteristics across three COVID-19-specific timepoints: during lockdown, immediately following lockdown, and three months after lockdown ended. We demonstrate that there are three unique subgroups of university students who access mental health apps. Two of these, with either higher or lower mental well-being, were defined by characteristics that were stable across COVID-19 timepoints. The third, situational well-being, had characteristics that were timepoint-dependent, suggesting that they are highly influenced by traumatic stressors and stressful situations. This subgroup also showed feelings and behaviours consistent with burnout. Overall, our findings clearly suggest that user subgroups are unique: they have different characteristics and therefore likely have different mental healthcare goals. Our findings also highlight the importance of including questions and additional interventions targeting traumatic stress(ors), reason(s) for use, and burnout in JITAI-style mental health apps to improve engagement.
BibTeX
@misc{shvetcov2022machine,
  author = {Artur Shvetcov and Alexis E. Whitton and Suranga Kasturi and Wu-Yi Zheng and Joanne R. Beames and Omar Ibrahim and Jin Han and Leonard Hoon and Kon Mouzakis and Sunil Gupta and Svetha Venkatesh and Helen Christensen and Jill M. Newby},
  title = {{Machine learning identifies a COVID-19-specific phenotype in university students using a mental health app}},
  year = {2022},
  doi = {10.1101/2022.12.07.22283234},
  howpublished = {medRxiv},
}
Figure