Abstract
Blood glucose prediction is a fundamental part of advanced technology that promises to improve diabetes outcomes. However, a critical gap exists around understanding the reproducibility of state-of-the-art methods for blood glucose prediction. In this study, we curated 60 deep learning (DL)-based glucose prediction papers published between 2018–2025 and assessed them against seven established reproducibility criteria. We found that code availability, overreliance on a single public dataset, and limited use of multiple datasets for algorithm development and evaluation are amongst the top challenges to reproducibility. Next, we replicated six representative models from well-cited prior literature using 4602 days of data with over 1.25 million glucose samples from 128 persons with type 1 diabetes across three public datasets - OhioT1DM, DiaTrend, and T1DEXI. Our results show good reproducibility of DL methods when using the same code (where available) and same evaluation dataset. However, we found poor conceptual reproducibility across datasets with significantly different diabetes management. Further analyses revealed that the accuracy of blood glucose prediction methods was significantly associated with individual diabetes management and sex/gender. All models had significantly higher prediction errors for individuals with worse glycemic control and for female subgroups compared to males. To accelerate development of robust and equitable algorithms for diabetes management, we conclude with recommendations for future researchers centered considerations for data selection, model design and selection, model evaluation and reporting results, documentation and code release.
Introduction
Managing Type 1 Diabetes (T1D) is a relentless, 24/7 task of balancing insulin, food, and activity to maintain blood glucose (BG) within a safe range. Deep learning (DL) has emerged as a powerful tool for predicting future BG levels, offering the potential to prevent dangerous hypoglycemic or hyperglycemic events. However, the field faces a "reproducibility crisis." While hundreds of papers claim state-of-the-art results, few share their code, and most rely on a single, small public dataset (OhioT1DM). This study audits the reproducibility of these models and investigates whether they work equitably across diverse populations.
Systematic Review of Reproducibility
The researchers curated 60 DL-based glucose prediction papers published between 2018 and 2025, scoring them against seven reproducibility criteria. The review revealed significant gaps in the field's transparency.
As shown in Figure 1, only 21.7% of the reviewed papers made their code publicly available, creating a major roadblock for technical reproducibility. Furthermore, 55% of studies relied exclusively on the OhioT1DM dataset, which, while a benchmark, is limited in size (n=12). This overreliance suggests that many models may be optimized for a narrow slice of the population rather than general physiological dynamics.
Replication and the Generalization Wall
To test these findings, the researchers selected six representative models (LSTMs, CNNs, and Transformers) and evaluated them across three public datasets: OhioT1DM (benchmark), DiaTrend (larger, more variable), and T1DEXI (diverse insulin delivery modes).
The replication study (Table 1) demonstrated that while "technical reproducibility" (same data/code) was generally successful, "conceptual reproducibility" (different datasets) was poor. Models that performed excellently on OhioT1DM saw their Root Mean Square Error (RMSE) jump significantly when tested on the DiaTrend dataset. This "generalization wall" indicates that current models often overfit to specific dataset characteristics rather than learning the underlying physiology of glucose dynamics.
The Equity Gap and Predictability
A striking finding of the study is the significant performance gap linked to sex and glycemic control. All six replicated models performed worse for female subgroups compared to males, with an average error difference of 2.21 mg/dL (p < 0.05). This bias was not explained by differences in diabetes management, suggesting a need for sex-specific model considerations.
Furthermore, predictability appears to be a "privilege." There is a strong correlation between a patient's glycemic control and model accuracy; individuals with higher HbA1c or lower "Time in Range" (TIR) experienced significantly higher prediction errors. Ironically, those who need accurate predictions the most are currently the least served by these algorithms.
Conclusion and Recommendations
To move toward robust and equitable digital health, the authors recommend:
- Standardized Reporting: Researchers should report results across multiple datasets and provide detailed population distributions.
- Open Science: Releasing full replication code and dataset processing scripts is essential for progress.
- Personalized Models: Exploring sex-specific or management-style-specific models (e.g., MDI vs. Pump) to close the equity gap.
By addressing these reproducibility and equity challenges, the field can develop algorithms that truly improve outcomes for all individuals living with diabetes.
Summary updated