Predicting Long-Term Engagement in mHealth Apps: Comparative Study of Engagement Indices

Yae Won Tak, Jong Won Lee, Junetae Kim, Yura LeeJournal of Medical Internet Research 2024 · DOI

TL;DR

This paper introduces a novel engagement index (EI) for mHealth apps that significantly improves the prediction of long-term user engagement and compliance compared to existing methods. By leveraging detailed app usage logs, this new index offers a more robust and objective measure, crucial for understanding and optimizing digital health interventions.

What Problem Does This Paper Address?

Digital health therapeutics (DTx) and mHealth apps hold immense promise for improving healthcare accessibility and patient engagement. However, a significant challenge in this field is the lack of standardized, quantitative tools to measure long-term user engagement and adherence. Current methods often rely on subjective post-intervention surveys or interviews, which are time-consuming, prone to bias, and may not accurately reflect sustained usage patterns. This gap hinders our ability to effectively evaluate the impact and optimize the design of these digital health solutions.

Methods

The study proposes and evaluates a new engagement index (EI) for mHealth apps, building upon existing frameworks like the Web Matrix Visitor Index. The core of their approach involves analyzing detailed app usage log data.

Key methodological aspects include:

  • Existing EI: The study first evaluates a baseline EI, which is an average of three indices: click depth, recency, and loyalty.
  • New Engagement Index (EI): The authors developed a novel EI by refining and introducing new components:
  • Menu Abundancy Index (MI): This new index considers the variety of menus available within the app and the "survival time" of each menu (time elapsed before a menu is not used). It quantifies engagement based on the Euclidean distance from a baseline user.
  • Enriched Loyalty Index (LI): The traditional loyalty index (based on the number of accessed weeks) is enhanced by incorporating the final usage week, with a simple linear transformation to keep values between 0 and 1.
  • Refined Recency Index (RI): This index measures the regularity of app usage using permutation entropy (PE), a robust time series tool that quantifies the complexity of usage patterns. A higher PE value indicates less regularity.
  • Prediction Models:
    • Multiple Linear Regression: Used to predict the EI at a later time point (6-9 months) based on the EI at an earlier time point (3-6 months).
  • Survival Analysis (Cox Regression): Used to predict the survival rate (i.e., continued app usage) from the 3-6 month period.
  • Data Analysis: All analyses were conducted using Python.

Datasets

The study utilized data from 233 cancer survivors (breast, colorectal, or lung cancer) who participated in a randomized controlled trial (RCT) evaluating the impact of mHealth apps on recovery. These participants were part of a larger RCT involving 960 individuals. The data was collected from the Noom app, a commercially available weight management tool, and included detailed usage logs. No new benchmark datasets were introduced; instead, existing app usage data was leveraged.

Key Insights and Ideas

  • Log Data is King: The study strongly emphasizes the importance of objective log data for measuring mHealth engagement, moving beyond subjective self-reports.
  • Beyond Simple Metrics: The new EI, particularly with the inclusion of Menu Abundancy and refined Recency/Loyalty indices, offers a more nuanced understanding of user interaction than simpler metrics.
  • Predictive Power Matters: The core insight is that a well-constructed EI can effectively predict future engagement and compliance, which is crucial for intervention success.
  • Menu Variety and Regularity are Key: The new EI's components, especially MI and PE-based RI, highlight that not just what users do, but how often and how varied their interactions are, significantly impacts long-term engagement.
  • Subjectivity Can Be Mitigated: While acknowledging the inherent subjectivity in some EI metrics, the study demonstrates how objective log data and refined calculations can reduce bias.

Key Results

  • Improved Prediction: The newly developed EI model significantly outperformed the existing EI in predicting 6-9 month engagement based on 3-6 month data.
  • Existing EI: MSE of 0.096, RMSE of 0.310, R² of 0.053.
  • New EI (best model): MSE of 0.025, RMSE of 0.157, R² of 0.610.
  • Survival Rate Prediction: The new EI also showed better predictive power for survival rates.
  • Existing EI indices: Click depth (HR 0.49, P<.001) and Loyalty (HR 0.17, P<.001) were significant predictors of improved survival. Recency was not.
  • New EI models: Showed significant associations with Loyalty (HR 0.32, P<.001) and Recency (HR 0.47, P<.001) for the best performing new EI.
  • Demographics: The study cohort consisted of 233 participants, with an average age of 53.55 years, predominantly women, with various cancer types and stages.

Limitations and Future Work

  • Limitations:
    • Subjectivity: Despite improvements, the EI still has a subjective nature that may introduce biases.
  • Limited Indices: The evaluation of the existing EI was limited to 3 indices (click depth, recency, loyalty) due to app-specific data availability.
  • Simplicity vs. Effectiveness: While the existing EI might be simpler to calculate, the new EI is deemed more effective for prediction.
  • Data Availability: The raw data sets are not publicly available due to IRB approval and privacy concerns.
  • Future Work:
    • Address Subjectivity: Further research is needed to address the inherent subjectivity in EI metrics.
  • Broader Indices: Incorporate a wider range of indices for a more comprehensive evaluation.
  • Generalizability: While the study focused on the Noom app, future work could explore the generalizability of the new EI across different mHealth platforms and user populations.

Summary updated

Abstract

The newly developed EI model outperformed the existing model in terms of the prediction of long-term user engagement and compliance in a mobile health app context. We emphasized the importance of log data and suggested avenues for future research to address the subjectivity of the EI and incorporate a broader range of indices for comprehensive evaluation.
BibTeX
@article{tak2024predicting,
  author = {Yae Won Tak and Jong Won Lee and Junetae Kim and Yura Lee},
  title = {{Predicting Long-Term Engagement in mHealth Apps: Comparative Study of Engagement Indices}},
  journal = {Journal of Medical Internet Research},
  year = {2024},
  doi = {10.2196/59444},
}
Figure