1294 External validation of machine learning models to predict efficacy and toxicity of immune checkpoint inhibitors using real-world pan cancer cohorts

Zoltán Kiss, Levente Lippenszky, Balázs Laczi, Pablo Napan-Molina, Eszter Csernai, Alexander Brehmer, Moon Kim, Julius Keyl, Jens T. Siveke, Moritz Meyer, Viktor Grünwald, Stefan Kasper, Alexander Roesch, Martin Schüler, Travis Osterman, Jan Wolber, Jens KleesiekRegular 2023 · DOI

Introduction

Immune checkpoint inhibitors (ICIs) have transformed the landscape of oncology, offering significant survival benefits for patients with various malignancies. However, the clinical response to ICIs is highly variable, and treatment is often associated with immune-related adverse events (irAEs), such as drug-induced hepatitis. Predicting which patients will benefit from therapy and which are at risk for toxicity is a critical challenge in personalized medicine.

Machine learning (ML) offers a powerful approach to address this by analyzing complex, multi-dimensional patient data. Nevertheless, most ML models in this space are developed using single-center data, raising concerns about their generalizability across different clinical environments. This study presents a large-scale external validation of an ML framework trained on US clinical data and evaluated on a European cohort, addressing the critical challenge of cross-institutional generalizability.

Methods

The researchers developed Random Forest classification models to predict two primary clinical endpoints:

  1. Efficacy: Overall Survival (OS) at 100 days, one year, and three years post-treatment initiation.
  2. Toxicity: Occurrence of hepatitis (defined as liver enzymes exceeding 3x the upper limit of normal) within six weeks, 90 days, and one year.

The models utilized structured, routinely available real-world data (RWD). Feature engineering involved aggregating laboratory measurements over a 120-day window to capture temporal trends, while a one-year lookback window was applied for other data types such as medication history and condition codes.

Table 1: Overview of features, targets, and data processing for the Hepatitis and Overall Survival (OS) models.

To facilitate cross-institutional data integration, the team used FHIRPACK, a tool designed for standardized data extraction from FHIR (Fast Healthcare Interoperability Resources) databases. The models were containerized to ensure consistent performance and ease of deployment across different IT infrastructures.

Datasets

The study utilized two large-scale, pan-cancer cohorts consisting of patients with both localized and metastatic tumors:

  • Training Cohort: Data from Vanderbilt University Medical Center (VUMC), USA, including over 2,200 patients.
  • External Validation Cohort: A retrospective cohort from the University Hospital Essen, Germany, consisting of 4,257 patients.

The use of an external validation cohort nearly twice the size of the training set provides a rigorous test of the model's robustness and statistical reliability.

Key Results

The Random Forest models demonstrated high predictive accuracy in the training phase and, crucially, maintained strong performance when applied to the external German cohort, suggesting that real-world clinical data is consistent enough for cross-border applications.

Overall Survival (OS) Prediction

In the training set, the models achieved an Area Under the Curve (AUC) of up to 0.79 for OS. Upon external validation, the models retained over 90% of this performance. Specifically, the 100-day OS model achieved an AUC of 0.77 in the German cohort.

Table 2: Performance metrics (AUC) for Overall Survival (OS) models across training and external validation cohorts.

Hepatitis (Toxicity) Prediction

For the prediction of hepatitis, the training AUC reached 0.81. The models retained 86% of their performance on the external cohort, with the 6-week toxicity model achieving an AUC of 0.75.

The results indicate that short-term outcomes (e.g., 100-day survival or 6-week toxicity) are more accurately predicted than long-term ones. This suggests that the immediate physiological state of the patient is a strong indicator of early treatment response, whereas long-term outcomes are likely influenced by additional confounding factors such as subsequent lines of therapy or disease progression.

Discussion and Conclusion

This study represents one of the first large-scale, cross-continental external evaluations of ML models for predicting ICI outcomes. The successful validation of models trained in the US on a German cohort demonstrates that real-world clinical data is sufficiently consistent to support the development of generalizable predictive tools.

The use of standardized data formats like FHIR and containerized deployment strategies highlights a path forward for integrating AI into routine clinical practice. These models could serve as valuable clinical decision support tools, helping clinicians pre-screen patients for ICI clinical trials and manage treatment risks more effectively. Future work should focus on prospective validation and the incorporation of longitudinal data to improve long-term predictive accuracy.

Summary updated

Abstract

<h3>Background</h3> Predicting a patient‘s response to immune checkpoint inhibitors (ICIs) could help understand the benefit-risk profile of treatment, potentially improve clinical trial cohort selection, and may inform care pathway decisions in clinical practice. Recently, machine learning (ML)-based predictive analytics have gained momentum in this area, but models trained or evaluated on multi-center data are still rare. Therefore, it is difficult to assess the generalizability of single-center models. We present the results of an external validation of a ML framework trained on US data on a large European cohort. <h3>Methods</h3> Random forest classification models were built to predict overall survival (OS) greater than 100 days, one year, and three years, and to predict the occurrence of hepatitis within six weeks, 90 days, and one year after initiation of ICI treatment. For model training, we utilized routinely available real-world data from Vanderbilt University Medical Center (data cut-off December 31, 2018) of a more than 2,200 patient strong pan-cancer cohort containing patients with localized as well as metastatic tumors. Structured, routine clinical data such as age, laboratory values, medication history and condition codes were used as model features. Feature engineering involved aggregating laboratory measurements acquired over a 120-day time window, and a one-year window was applied for other data types. The hepatitis binary label was defined as 1 if any liver enzymes exceeded three times the upper limit of normal (table 1). The trained models were evaluated in an external retrospective pan-cancer cohort of the University Hospital Essen, Germany (n=4257). All input variables were extracted from a FHIR database using FHIRPACK.<sup>1</sup> Containerized models were employed for data integration and model evaluation. <h3>Results</h3> Our random forest models achieved an AUC of up to 0.79 for the prediction of OS and up to 0.81 for the prediction of hepatitis in the training data. The models successfully retained at least 90% of their performance for OS and 86% for hepatitis prediction endpoints on the external evaluation cohort (tables 2 and 3). <h3>Conclusions</h3> To our knowledge, this is the first large-scale, external cohort evaluations of OS and hepatitis prediction ML models in ICI patients. Despite different geographic origins, our models generalized well to unseen data. In particular, short-term models showed remarkable performance retention when applied to an external cohort. Our work demonstrates the potential of ML models as valuable tools for pre-screening eligible patients for ICI clinical trials and as clinical decision support for routine patient management. <h3>Reference</h3> https://github.com/fhirpack/fhirpack <h3>Ethics Approval</h3> Ethics approval for using the Vanderbilt university cohort was granted by The Vanderbilt University Medical Center Health Sciences #3 Institutional Review Board, tracked as #211814. The IRB determined the study poses minimal risk to participants, and a waiver of consent was granted. The study using the external validation cohort was approved by the Ethics Committee of the Medical Faculty of the University of Duisburg-Essen (No. 21–10347-BO). The requirement for written informed consent was waived due to the retrospective design of the study.
BibTeX
@inproceedings{kiss2023external,
  author = {Zoltán Kiss and Levente Lippenszky and Balázs Laczi and Pablo Napan-Molina and Eszter Csernai and Alexander Brehmer and Moon Kim and Julius Keyl and Jens T. Siveke and Moritz Meyer and Viktor Grünwald and Stefan Kasper and Alexander Roesch and Martin Schüler and Travis Osterman and Jan Wolber and Jens Kleesiek},
  title = {{1294 External validation of machine learning models to predict efficacy and toxicity of immune checkpoint inhibitors using real-world pan cancer cohorts}},
  booktitle = {Regular},
  year = {2023},
  doi = {10.1136/jitc-2023-sitc2023.1294},
}
Figure