Validating the early phototherapy prediction tool across cohorts

Imant Daunhawer, Kai Schumacher, Anna Badura, Julia E. Vogt, Holger Michel, Sven WellmannFrontiers in Pediatrics 2023 · DOI

Abstract: The individual risk for treatment requirement in neonatal hyperbilirubinemia is robustly predictable in different patient cohorts with a previously developed ML tool (EPPT) demanding just one TSB value and only four clinical parameters. Further prospective validation studies are needed to develop an effective and safe clinical decision support system.

Introduction

Neonatal hyperbilirubinemia, commonly known as jaundice, remains the leading cause of hospital readmission during a newborn's first month of life. While typically benign, extreme cases can lead to irreversible neurological damage if not treated promptly with phototherapy. Traditional clinical management relies on static nomograms—risk-stratification charts that often lack personalization and necessitate frequent, invasive blood sampling.

To address this, researchers previously developed the Early Phototherapy Prediction Tool (EPPT), a machine learning ensemble designed to provide individualized risk scores. However, a major hurdle for clinical AI is "distribution shift": models often fail when applied to new hospital populations with different demographics or treatment protocols. This study evaluates the EPPT's robustness by validating it against a diverse new cohort, testing its readiness for real-world clinical environments.

Methodology and Variable Importance

The EPPT utilizes an ensemble machine learning approach, combining LASSO (Regularized Logistic Regression) and Random Forest models. LASSO identifies strong linear relationships and prevents overfitting, while the Random Forest captures complex, non-linear interactions between variables.

The tool generates a probability score for whether a neonate will require phototherapy within the next 48 hours. This calculation is based on five primary clinical variables:

  1. Total Serum Bilirubin (TSB) value
  2. Gestational age at birth
  3. Birth weight
  4. Postnatal age (hours since birth)
  5. Bilirubin-to-weight ratio
Relative importance of clinical variables in the EPPT model, highlighting gestational age and weight as primary predictors alongside bilirubin levels.

Datasets and External Validation

The study compared the performance of the original model against a version re-trained on a new dataset:

  • Original Training Set (UKBB): 362 newborns from University Children’s Hospital Basel, Switzerland.
  • Validation Set (KUNO): 1,109 neonates (3,940 individual measurements) from University Children’s Hospital Regensburg, Germany.

The KUNO cohort presented a rigorous test for external validity, as it contained significantly fewer preterm infants and operated under different local phototherapy guidelines than the original Swiss cohort.

Key Results

The validation results demonstrate that the EPPT is remarkably robust against shifts in patient populations and clinical guidelines. The "off-the-shelf" original model achieved an 84.6% AUROC on the German cohort without any local modifications. When the model was re-trained on a subset of the new data, performance improved to an 88.8% AUROC.

ROC curves comparing the performance of the original Swiss-trained model and the re-trained German-cohort model on the validation dataset.

On a patient level, the model maintained a high specificity of 90.7%. It correctly predicted 77 out of 239 phototherapy instances within the 48-hour window while keeping false-positive rates low, which is critical for minimizing unnecessary medical interventions.

Discussion and Future Directions

A key insight from the study was feature parsimony: adding additional variables like delivery mode or maternal blood group did not significantly enhance predictive power. The original five variables proved sufficient for high-accuracy forecasting. Furthermore, the model showed high data efficiency, reaching near-peak performance even when trained on only 25% of the available data, suggesting that smaller hospitals could successfully fine-tune the tool for their specific populations.

While these retrospective results are promising, the authors emphasize that prospective validation is the next necessary step. Future research will determine if using the EPPT in real-time actually improves clinical outcomes, reduces the frequency of blood draws, or can be adapted for non-invasive transcutaneous measurements. Expanding the model to predict treatment duration and the risk of "rebound" jaundice also remains a priority for future development.

Summary updated

Abstract

The individual risk for treatment requirement in neonatal hyperbilirubinemia is robustly predictable in different patient cohorts with a previously developed ML tool (EPPT) demanding just one TSB value and only four clinical parameters. Further prospective validation studies are needed to develop an effective and safe clinical decision support system.
BibTeX
@inproceedings{daunhawer2023validating,
  author = {Imant Daunhawer and Kai Schumacher and Anna Badura and Julia E. Vogt and Holger Michel and Sven Wellmann},
  title = {{Validating the early phototherapy prediction tool across cohorts}},
  booktitle = {Frontiers in Pediatrics},
  year = {2023},
  doi = {10.3389/fped.2023.1229462},
}
Figure