Recruiting Young People for Digital Mental Health Research: Lessons From an AI-Driven Adaptive Trial

Wu Yi Zheng, Artur Shvetcov, Aimy Slade, Zoë Jenkins, Leonard Hoon, Alexis E. Whitton, Rena Logothetis, Smrithi Ravindra, Stefanus Kurniawan, Sunil Gupta, Kit Huckvale, Eileen Stech, Akash Agarwal, Joost Funke Kupper, Stuart Cameron, Jodie Rosenberg, Nicholas Manoglou, Manisha Senadeera, Svetha Venkatesh, Kon Mouzakis, Rajesh Vasa, Helen Christensen, Jill M. NewbyJournal of Medical Internet Research 2024 · DOI

TL;DR

Recruiting young adults for digital mental health trials is challenging, especially with AI-driven adaptive designs requiring frequent enrollment. This study found that social media advertising (Facebook and Instagram) was the most effective and cost-efficient method for recruiting participants, even outperforming traditional methods. While social media excelled at recruitment, no single method stood out for participant retention.

What Problem Does This Paper Address?

The increasing adoption of remote clinical trials, particularly in digital mental health, necessitates efficient and cost-effective recruitment strategies. This is especially true for AI-driven adaptive trials, which require continuous enrollment over shorter periods compared to traditional randomized controlled trials (RCTs). While web-based recruitment methods are gaining traction, evidence on their effectiveness relative to traditional methods (like newspapers or flyers) is inconsistent. Furthermore, the COVID-19 pandemic introduced unique challenges and opportunities for recruitment, potentially altering cost-effectiveness. This paper aims to address these gaps by evaluating the effectiveness and cost-effectiveness of various recruitment methods, including social media, for a digital mental health trial targeting young adults, with a specific focus on the impact of COVID-19 lockdowns.

Methods

The study employed a mixed-methods approach, analyzing recruitment data from the "Vibe Up" AI-driven adaptive digital mental health trial. Recruitment occurred between June 2021 and May 2022, encompassing two pilot trials and six "mini-trials."

Key recruitment strategies included:

  • Paid social media advertising: Utilizing Facebook and Instagram with "dynamic creative" features to optimize ad combinations.
  • Traditional methods: Employing university mailing lists, student organizations, media releases, in-class announcements, study posters/flyers on campuses, and health professional networks.

Data analysis involved:

  • Meta (Facebook and Instagram) analytics: To collect engagement metrics and advertising costs.
  • Qualtrics survey data: To track recruitment methods, screening, eligibility, and survey completion rates.
  • RStudio with R: For statistical analysis, including descriptive statistics, general linear models with pairwise post hoc tests, Fischer exact tests, Kruskal-Wallis tests, and Wilcoxon tests for cost-effectiveness comparisons.

The study also specifically compared the cost-effectiveness of social media recruitment during and after COVID-19 lockdowns.

Datasets

The study utilized data collected from participants in the "Vibe Up" trial. This included:

  • Recruitment data: Information on how participants were recruited, their engagement with advertisements, and screening/eligibility status.
  • Advertising cost data: Expenses incurred for Facebook and Instagram advertising campaigns.
  • Qualtrics survey data: Responses related to recruitment methods and survey completion.

No new datasets were introduced; the study analyzed data generated during the trial.

Key Insights and Ideas

  • Social Media Dominance: Facebook and Instagram advertising were overwhelmingly the most successful channels for recruiting young adults into this digital mental health trial, accounting for 92% of eligible participants. This highlights the power of these platforms for reaching this demographic.
  • Website's Niche Effectiveness: While social media brought in the most participants overall, the lead institution's website was found to be more effective at converting screened individuals into eligible participants. This suggests that individuals actively seeking mental health resources might be more likely to engage with information presented on a trusted institutional site.
  • Lockdown's Cost-Effectiveness: Social media recruitment was significantly more cost-effective during the COVID-19 lockdowns compared to the post-lockdown period. This was observed in both the cost per link click and the amount spent per hour to reach the target sample size. This suggests that periods of increased distress or reduced external activity might make individuals more receptive to online interventions and recruitment.
  • Retention is a Separate Challenge: No single recruitment method demonstrated a significant advantage in participant retention from baseline to study completion. This indicates that while social media is excellent for initial recruitment, other factors likely influence long-term engagement.
  • Dynamic Creative and Pixel Optimization: The study highlighted the importance of optimizing social media ad strategies. Initially using a "link click" objective, they later shifted to a "landing page view" and then a "conversion pixel" ("GET STARTED" button) for subsequent trials, demonstrating how fine-tuning ad objectives can improve efficiency and target more engaged users.

Key Results

  • Participant Recruitment: A total of 1314 eligible participants were recruited.
  • 92% (n=1203) were recruited via Facebook and Instagram advertising.
  • The lead institution's website recruited 4% of eligible participants.
  • Cost-Effectiveness:
    • Average cost per screened participant: Aus $2.28 (US $1.42).
  • Average cost per eligible participant: Aus $6.59 (US $4.10).
  • Average cost per app download: Aus $7.78 (US $4.84).
  • Average cost per trial completion: Aus $10.01 (US $6.22).
  • COVID-19 Lockdown Impact:
    • Cost per link click was significantly lower during lockdown (median Aus $0.35) compared to post-lockdown (median Aus $1.00).
  • Cost per hour to reach the target sample size was also significantly lower during lockdown (median Aus $4.75) compared to post-lockdown (median Aus $13.29).
  • Social Media Engagement:
    • Over 1.1 million impressions were generated.
  • 825,635 people were reached.
  • 8138 clicks were registered on the study website, with 7858 unique link clicks.
  • Overall click-through rate was 0.73%.

Limitations and Future Work

  • Cost-Effectiveness of Traditional Methods: The study could not assign monetary values to traditional recruitment methods (posters, flyers, announcements), preventing direct cost-effectiveness comparisons. Future work could attempt to quantify these costs.
  • Sex Imbalance: Despite efforts, a sex imbalance (predominantly female participants) persisted. The authors suggest user testing of tailored advertisements for male participants in future studies.
  • Lockdown Categorization: Due to varying lockdown periods across Australia, trials were categorized based on published government dates, which might not perfectly reflect individual participant lockdown status.
  • Generalizability: The study focused on tertiary education students. Findings may not be generalizable to all young people, particularly those not in higher education. Future research could explore recruitment strategies for broader young adult populations.
  • Retention Strategies: No recruitment method stood out for retention. Future work should investigate strategies specifically aimed at improving participant engagement and retention in digital mental health trials.
  • International Student Recruitment: The authors suggest targeted social media advertising and liaising with student leaders to improve recruitment of international students.

Summary updated

Abstract

Australian New Zealand Clinical Trials Registry ACTRN12621001092886; https://tinyurl.com/39f2pdmd; Australian New Zealand Clinical Trials Registry ACTRN12621001223820; https://tinyurl.com/bdhkvucv.
BibTeX
@article{zheng2024recruiting,
  author = {Wu Yi Zheng and Artur Shvetcov and Aimy Slade and Zoë Jenkins and Leonard Hoon and Alexis E. Whitton and Rena Logothetis and Smrithi Ravindra and Stefanus Kurniawan and Sunil Gupta and Kit Huckvale and Eileen Stech and Akash Agarwal and Joost Funke Kupper and Stuart Cameron and Jodie Rosenberg and Nicholas Manoglou and Manisha Senadeera and Svetha Venkatesh and Kon Mouzakis and Rajesh Vasa and Helen Christensen and Jill M. Newby},
  title = {{Recruiting Young People for Digital Mental Health Research: Lessons From an AI-Driven Adaptive Trial}},
  journal = {Journal of Medical Internet Research},
  year = {2024},
  doi = {10.2196/60413},
}
Figure