Introduction
Digital health interventions (DHIs), such as mobile applications for physical activity or substance use management, are increasingly utilizing online AI. Unlike "offline" models that are trained on static datasets, online AI systems learn and adapt in real-time based on continuous data streams. While this allows for personalized care, it introduces significant challenges for scientific reproducibility.
The core issue lies in the adaptive feedback loop: the AI's decisions influence user behavior, which subsequently alters the data the AI receives. This circular dependency can lead to "divergence" where small variations in code or data latency result in vastly different system states.
The Reproducibility Crisis in Digital Health
The authors identify two primary sources of instability that hinder reproducibility:
- Engineering Instability: Discrepancies between development and production environments, inconsistent codebases, or silent system failures.
- Algorithmic Instability: Learning algorithms that lack convergence guarantees or react unpredictably to noisy, real-world data.
Without a reproducible framework, researchers cannot audit specific AI decisions, compromising both scientific discovery and patient safety.
A Three-Phase Reproducible Workflow
To address these challenges, a systems-engineering approach is proposed, dividing the AI lifecycle into three distinct phases.
Phase 1: Design Reproducibility
Before deployment, candidate algorithms are stress-tested using Digital Twins. These are high-fidelity simulators constructed from historical data that allow researchers to evaluate how an algorithm handles user heterogeneity, data drift, and missing values in a controlled "sandbox" environment.
Phase 2: Deployment Reproducibility
This phase focuses on the technical infrastructure required for reliable real-time operation:
- Software Isolation: Utilizing containerization (e.g., Docker) and orchestration (e.g., Kubernetes) ensures that the execution environment remains identical across all stages of the project.
- Traceable Logging: Every input, model state, and random seed must be recorded. This creates an audit trail that allows for a post-hoc "replay" of the AI's logic to verify its decisions.
Phase 3: Inference Reproducibility
Post-deployment, the logged data is used for rigorous performance analysis. These insights are then fed back into Phase 1 to inform the next iteration of the algorithm design.
Lessons from the Field: The "Imputation Trap"
The framework is grounded in real-world interventions like HeartSteps (physical activity), Oralytics (oral care), and MiWaves (cannabis use reduction). A key insight from the HeartSteps study is the Imputation Trap. Due to network delays, the AI often had to use estimated (imputed) step counts to make immediate decisions. When the actual data arrived later, the system would overwrite the imputed values. This practice destroyed the record of what the AI actually "saw" at the moment of decision, making it impossible to reconstruct the logic behind specific interventions.
Recommendation: Never overwrite real-time decision data with delayed ground truth; maintain both for a complete audit trail.
Operational Recommendations for Robust AI
- Strict Separation: Maintain a hard line between development and production to avoid "dependency hell."
- Fallback Mechanisms: Define "safety net" behaviors (e.g., default or random interventions) to ensure patient care continues even if the AI system crashes or data is missing.
- Validation via Replay: Researchers should be able to input logged data back into the algorithm to see if the resulting model states match the original logs.
Conclusion and Future Directions
While this workflow provides a roadmap for reproducibility, challenges remain. The reliance on proprietary hardware (e.g., smartwatches) often introduces "black box" data processing that researchers cannot control. Furthermore, resource constraints on wearable devices can limit the depth of data logging. Future research should focus on modular Digital Twins and stable autonomous learning rules to further bridge the gap between AI theory and clinical practice.
Summary updated