TL;DR
Clinical prediction models degrade over time as the real world changes. This paper explores "dynamic updating" – repeatedly refreshing models with new data – for survival prediction models, finding it generally outperforms one-time updates. Bayesian dynamic updating shows particular promise, especially when new predictors emerge or data is scarce.
What Problem Does This Paper Address?
Clinical prediction models are invaluable tools, but their accuracy can decline as patient populations, disease prevalence, or treatment practices evolve. This phenomenon, known as "model drift," necessitates updating these models to maintain their usefulness. While discrete, one-time updates have been studied, this paper focuses on dynamic updating, a continuous process of refreshing models as new data becomes available. The authors are particularly interested in time-to-event (survival) prediction models, which are crucial for understanding patient prognosis over time, and were motivated by the rapidly changing landscape of the COVID-19 pandemic. They aim to compare different dynamic updating strategies against each other and against static (no update) approaches.
Methods
The study compares three primary methods for dynamic model updating:
- Intercept Recalibration: This method updates only the baseline hazard of a Cox proportional hazards model, keeping the original predictor coefficients fixed. It's computationally light and doesn't change the relative risk predictions between individuals.
- Refitting: This is the most extreme approach, where the entire model (coefficients and baseline hazard) is re-estimated using new data. It can easily incorporate new predictors but is prone to overfitting, especially with small datasets.
- Bayesian Dynamic Updating: This method combines prior knowledge (from the existing model) with new data using a Bayesian framework. It assumes an exponential baseline hazard and uses a "forgetting factor" to balance the influence of old and new data. This approach is more computationally intensive but offers smoother updates and can handle new predictors.
The researchers also compare these dynamic methods against a "no updating" strategy (using the original model unchanged) and one-time updates (refitting or recalibrating at a single point in time). Performance is evaluated using standard metrics for survival models: the C-index (for discrimination), calibration intercept and slope, and Brier score (for overall performance).
Datasets
The study utilizes two main data sources:
- Simulated Data: A simulation study was conducted to create controlled environments. This allowed the researchers to systematically evaluate the updating methods under various conditions, including changing mortality rates, rare predictors, and the introduction of new treatments. Two data generation frameworks were used: "cohort with replacement" (mimicking continuous patient follow-up with some churn) and "new cohorts" (representing distinct groups of individuals over time).
- QResearch Database: This is a large, anonymized UK electronic health records database. A subset of this data was used to apply the updating methods to a real-world problem: predicting 70-day COVID-19-related mortality. This dataset provided a practical, albeit complex, scenario with a rare outcome and a rapidly evolving environment.
Key Insights and Ideas
- Dynamic is Generally Better: The core takeaway is that continuously updating models as new data arrives (dynamic updating) generally leads to better performance than waiting for a significant drift and performing a single, discrete update.
- Bayesian Updating's Versatility: Bayesian dynamic updating emerged as a strong contender, particularly excelling in scenarios with new predictors and limited data. Its ability to smoothly incorporate new information without drastic changes in hazard ratios is a significant advantage.
- Recalibration's Simplicity and Effectiveness: Intercept recalibration is a surprisingly effective and computationally cheap method for maintaining calibration, especially when data is scarce or the baseline risk is changing. It's also user-friendly as it doesn't alter relative risk rankings.
- Refitting's Double-Edged Sword: While refitting can adapt to new predictors and environments, it's highly susceptible to overfitting, especially with smaller datasets. This can lead to unstable hazard ratio estimates and abrupt changes in predictions.
- The Challenge of Rare Outcomes: In the COVID-19 application, the rarity of the outcome and the dynamic nature of the pandemic made it difficult for any single updating method to consistently outperform others across all metrics and time points.
Key Results
- In simulations with changing mortality rates, all updating methods improved calibration compared to no updating.
- Bayesian updating consistently showed good performance, especially in scenarios with new predictors and limited data, often improving the C-index over no updating and refitting.
- In the COVID-19 example, no single updating method was universally best. However, Bayesian updating and refitting strategies generally produced better calibration intercepts than no updating or recalibration in certain periods.
- Refitting strategies often resulted in more variable hazard ratio estimates over time compared to Bayesian updating.
Limitations and Future Work
- Computational Cost: Bayesian dynamic updating, while effective, is computationally intensive, requiring significant processing time.
- Prior Specification: The performance of Bayesian updating can be sensitive to how prior information is obtained, especially when the original development dataset is unavailable. The study noted poor performance in the first Bayesian update due to a mismatch in model assumptions between the original and prior models.
- Rare Outcomes: The study highlights that for very rare outcomes, standard performance metrics like Brier scores might be less informative, and differences between methods can be small.
- Adaptive Updating: The authors suggest future work on adaptive strategies that trigger updates based on predefined performance thresholds or detect "concept drift" in the data.
- Vaccine Impact: They also propose updating COVID-19 models to include the impact of vaccines, a significant environmental change not fully captured in the study's timeframe.
- Clinical Input: The paper emphasizes the importance of retaining clinical judgment and not fully automating the updating process, as human expertise can identify trends before they are statistically evident.
Summary updated