Personalising monitoring for chemotherapy patients through predicting deterioration in renal and hepatic function

Pinkie Chambers, Matthew Watson, John Bridgewater, Martin Förster, Rebecca Roylance, Rebecca Burgoyne, Sebastian Masento, Luke Steventon, James H. King, Nick Duncan, Noura Al MoubayedCancer 2023 · DOI

TL;DR

This paper introduces a machine learning model to predict the risk of renal and hepatic function deterioration in chemotherapy patients. By accurately stratifying patients, the model aims to reduce unnecessary blood tests for low-risk individuals while ensuring closer monitoring for those at higher risk, ultimately improving patient safety and healthcare efficiency.

What Problem Does This Paper Address?

Chemotherapy, while a cornerstone of cancer treatment, carries a significant risk of toxicity due to its narrow therapeutic index and the variability in how patients metabolize drugs. This toxicity can manifest as impaired kidney (renal) and liver (hepatic) function. Current clinical practice involves routine blood tests for all patients before each chemotherapy cycle to monitor these functions. However, this approach is resource-intensive and may lead to unnecessary tests for many patients who are unlikely to experience significant changes. Conversely, relying solely on routine monitoring might miss subtle deteriorations in high-risk patients. The paper addresses the need for a more personalized and efficient monitoring strategy that can identify patients who truly require frequent testing, thereby optimizing resource allocation and enhancing patient safety.

Methods

The researchers developed and validated a Multi-Layer Perceptron (MLP) model, a type of deep learning neural network. They chose MLPs over traditional logistic regression models due to the multi-level nature of their dataset and the demonstrated superior performance of MLPs in similar applications. The model was trained using retrospective data from one hospital and validated on data from a separate, independent hospital.

Key methodological aspects include:

  • Data Preprocessing: Patient records were carefully curated, excluding those with incomplete data for key variables. The outcome of interest was defined as a grade change (using modified CTCAE guidelines) in creatinine or bilirubin following the second cycle of chemotherapy.
  • Feature Engineering: The model utilized routinely collected data from Electronic Prescribing (EP) systems, including baseline and early-cycle blood results (ANC, hemoglobin, creatinine, ALT, bilirubin), patient demographics (age, gender, ethnicity, height, weight), comorbidities, and treatment details (regimen, relative dose intensity).
  • Model Architecture: The MLP model consisted of an input layer, two hidden layers, and an output layer, employing the ReLU activation function. Dropout was used to prevent overfitting.
  • Validation Strategy: The model was internally validated using an 80-10-10 split (training, testing, validation) and externally validated on data from a different hospital. Ten-fold cross-validation was used to ensure a robust evaluation of the trained models.
  • Performance Metrics: Standard metrics like AUROC, F1 score, sensitivity, specificity, PPV, NPV, and Cohen's Kappa were used. Crucially, the paper emphasizes the importance of high sensitivity to minimize false negatives and compromise patient safety.
  • Clinical Utility Assessment: A Net Benefit analysis was performed to evaluate the clinical applicability of the model, comparing it against "treat all" (monitoring everyone) and "treat none" (monitoring no one) strategies.

Datasets

  • Training Dataset: Retrospective data from one specialist cancer hospital in the United Kingdom.
  • Validation Dataset: Retrospective data from a second, independent specialist cancer hospital in the United Kingdom.

The study included patients treated with chemotherapy for breast cancer, colorectal cancer, and diffuse large B-cell lymphoma. The paper does not introduce a new benchmark dataset but rather utilizes existing clinical data from two hospitals.

Key Insights and Ideas

  • The Power of Routine Data: The study demonstrates that common clinical data elements routinely collected in Electronic Prescribing (EP) systems are sufficient for building a powerful predictive model, negating the need for specialized or extensive data collection.
  • Balancing Safety and Efficiency: The core idea is to shift from a one-size-fits-all monitoring approach to a personalized one. The model's ability to identify low-risk patients allows for reduced testing, while its high sensitivity ensures that high-risk patients are not missed, thus enhancing safety.
  • The Trade-off for Safety: To minimize false negatives (missing patients who will deteriorate), the researchers intentionally adjusted CTCAE grade boundaries. This led to a higher number of false positives but was deemed clinically acceptable to ensure patient safety, even at the cost of a lower F1 score and PPV.
  • Clinical Utility Beyond Accuracy: The Net Benefit analysis is a crucial insight, showing that the model provides clinical value by offering a better strategy than either universal monitoring or no monitoring, across a range of probability thresholds.

Key Results

  • Patient Cohort: A total of 1214 patients were included across both hospitals (684 in hospital 1 for training, 530 in hospital 2 for validation).
  • Event Rates: 184 patients (15%) experienced a one-grade or greater change in creatinine or bilirubin following cycle 2. Creatinine changes were seen in 3% of patients, and bilirubin changes in 12%.
  • Model Performance (Validation Set):
    • Creatinine: AUROC of 0.76 (95% CI: 0.70–0.82), Sensitivity of 0.60 (95% CI: 0.55–0.64), Specificity of 0.98 (95% CI: 0.96–0.99).
  • Bilirubin: AUROC of 0.72 (95% CI: 0.68–0.76), Sensitivity of 0.54 (95% CI: 0.52–0.56), Specificity of 0.90 (95% CI: 0.87–0.94).
  • False Negatives: The study found low false negative rates, and importantly, all false negatives were only one grade change, with no impact on subsequent cycle dosing.
  • Net Benefit: The decision curve analysis indicated that the model provided greater net benefit than "treat all" or "treat none" strategies for both creatinine and bilirubin predictions.

Limitations and Future Work

  • Small Event Prevalence: The low occurrence of grade changes in creatinine and bilirubin is acknowledged as a limitation, making it challenging to achieve high PPV and potentially leading to overfitting. Further validation with larger sample sizes is needed.
  • Ethnicity Differences: Differences in ethnicity between the two hospitals were noted and may contribute to the slight reduction in model performance during validation.
  • Limited Data Scope: The model did not incorporate genetic sequencing data, cancer-specific biomarkers, or detailed cancer information beyond EP data.
  • Future Work:
    • Prospective Validation: The authors plan a prospective validation study across various first-line cancer treatments to assess economic costs and environmental impact.
  • Broader Applicability: Testing the model in different cancer types and treatments to extend its benefits.
  • Feature Selection: Investigating feature selection methods to apply the model in settings where not all data is collected.
  • Clinical Integration: Addressing the educational need for clinicians to understand and trust ML model outputs for rapid implementation.

Summary updated

Abstract

We have demonstrated that a MLP model can be used to reduce the number of blood tests required for some patients at low risk of organ dysfunction, whilst improving safety for others at high risk.
BibTeX
@inproceedings{chambers2023personalising,
  author = {Pinkie Chambers and Matthew Watson and John Bridgewater and Martin Förster and Rebecca Roylance and Rebecca Burgoyne and Sebastian Masento and Luke Steventon and James H. King and Nick Duncan and Noura Al Moubayed},
  title = {{Personalising monitoring for chemotherapy patients through predicting deterioration in renal and hepatic function}},
  booktitle = {Cancer},
  year = {2023},
  doi = {10.1002/cam4.6418},
}
Figure