Performance metrics for models designed to predict treatment effect

Carolien C. H. M. Maas, David M. Kent, Michael C. Hughes, Rommert Dekker, Hester F. Lingsma, David van KlaverenmedRxiv 2022 · DOI

TL;DR

Predicting how well a treatment will work for an individual patient is crucial but hard because we can't observe both outcomes. This paper introduces new metrics to evaluate how well models can predict this "treatment effect," extending existing methods for risk prediction. These new metrics, tested in simulations and a real-world dataset, help us better understand and compare models that aim to personalize treatment decisions.

What Problem Does This Paper Address?

In medicine, we often want to know not just if a treatment works on average for a population, but how well it will work for a specific patient. This is called predicting individualized treatment effect. The core challenge is that for any given patient, we can only observe the outcome of the treatment they actually received, not the outcome they would have had if they received an alternative treatment (the counterfactual outcome). This makes it difficult to evaluate models that predict these individual treatment effects. While metrics exist to assess how well models predict general risk (like the probability of developing a disease), they don't fully capture how well a model predicts the difference in outcomes between two treatments for an individual. This gap means we lack robust ways to compare and validate models that aim to personalize treatment.

Methods

The authors build upon existing work, particularly the "C-for-benefit" metric, which measures a model's ability to distinguish between patients who benefit from a treatment and those who don't (discriminative ability). They extend this by adapting established metrics for risk prediction – the E-statistics, cross-entropy, and Brier score – to the context of treatment effect prediction.

Here's how they adapted them:

  • Observed Pairwise Treatment Effect: Instead of individual outcomes, they look at pairs of similar patients who received different treatments. The "observed pairwise treatment effect" is the difference in their outcomes. This is a clever way to get a measurable outcome related to treatment effect.
  • Calibration Metrics (E-statistics): They redefined E-statistics (like E_avg-for-benefit, E50-for-benefit, E90-for-benefit) to measure the agreement between predicted and observed treatment effects. These metrics quantify how well the model's predictions align with the actual differences observed in matched pairs.
  • Overall Performance Metrics (Cross-entropy and Brier Score): They adapted the cross-entropy and Brier scores to assess the overall performance of treatment effect predictions. These metrics consider both calibration and discrimination simultaneously, providing a single score for how good a model is at predicting treatment effects.

They validated these metrics through:

  1. Simulation Study: They created an "optimal model" and then deliberately "perturbed" it in three ways to simulate models that overestimate average treatment effect, risk heterogeneity, or treatment effect heterogeneity. They then compared the performance metrics of these perturbed models against the optimal model.
  2. Case Study: They applied the proposed metrics to data from the Diabetes Prevention Program (DPP) using three different modeling approaches: a risk model, an effect model, and a causal forest.

Datasets

The primary dataset used for the case study is the Diabetes Prevention Program (DPP) data. This dataset includes participants at risk for developing diabetes, randomized to lifestyle intervention, metformin, or placebo. The authors used this data to illustrate the application of their proposed performance metrics.

The simulation study used a synthetic dataset generated based on the characteristics of the DPP data, allowing for controlled evaluation of the proposed metrics under various model assumptions.

Key Insights and Ideas

  • Bridging the Gap: The most significant contribution is extending well-understood risk prediction metrics to the more complex domain of individualized treatment effect prediction. This provides a much-needed toolkit for researchers and clinicians.
  • "Perturbed" Models Show Worse Performance: The simulation study clearly demonstrated that the proposed metrics can distinguish between a well-performing "optimal" model and "perturbed" models that make systematic errors in predicting treatment effects. This is crucial for validating the metrics themselves.
  • Calibration vs. Discrimination Trade-off: The case study hinted at a potential trade-off between calibration and discrimination in treatment effect estimation, suggesting that models that are better calibrated might be slightly worse at discriminating between patients with small and large treatment effects, and vice-versa. However, the authors note that the sample size in the case study was limited, so strict conclusions cannot be drawn.

Key Results

  • Simulation Study: The "perturbed models" consistently showed worse performance across all proposed metrics compared to the "optimal model." For example, E_avg-for-benefit was significantly higher for perturbed models (≥0.070) than the optimal model (0.001), and cross-entropy-for-benefit was also higher (≥0.757 vs. 0.733).
  • Case Study (DPP Data):
    • The three modeling approaches (risk model, effect model, causal forest) showed numerically similar performance across the proposed metrics, though some differences were observed.
  • The "effect model" and "causal forest" generally showed better calibration metrics than the "risk model."
    • The "causal forest" demonstrated numerically better discriminative ability and overall performance than the "risk model."
    • Confidence intervals for the performance metrics in the case study were often overlapping, indicating the need for larger sample sizes for definitive conclusions in real-world settings.

Limitations and Future Work

  • Limited Sample Size in Case Study: The authors acknowledge that the case study's sample size was limited, which led to overlapping confidence intervals for the performance metrics. This made it difficult to draw strong conclusions about the superiority of one model over another in that specific instance. They emphasize that heterogeneous treatment effect analyses generally require much larger sample sizes.
  • Binary Outcomes: The proposed metrics were developed for binary outcomes. The authors suggest that extending these metrics to continuous outcomes would be a valuable future research direction.
  • Trade-off Exploration: Further research is needed to explore the observed trade-off between calibration and discrimination in more detail, potentially with larger datasets.
  • Updating Strategies: The authors suggest that in future research, updating strategies could be considered if the proposed calibration metrics indicate miscalibration of treatment effect predictions.

Summary updated

Abstract

ABSTRACT Background Measuring the performance of models that predict individualized treatment effect is challenging because the outcomes of two alternative treatments are inherently unobservable in one patient. The C-for-benefit was proposed to measure discriminative ability. However, measures of calibration and overall performance are still lacking. We aimed to propose metrics of calibration and overall performance for models predicting treatment effect. Methods Similar to the previously proposed C-for-benefit, we defined observed pairwise treatment effect as the difference between outcomes in pairs of matched patients with different treatment assignment. We redefined the E-statistics, the cross-entropy, and the Brier score into metrics for measuring a model’s ability to predict treatment effect. In a simulation study, the metric values of deliberately “perturbed models” were compared to those of the data-generating model, i.e., “optimal model”. To illustrate these performance metrics, different modeling approaches for predicting treatment effect are applied to the data of the Diabetes Prevention Program: 1) a risk modelling approach with restricted cubic splines; 2) an effect modelling approach including penalized treatment interactions; and 3) the causal forest. Results As desired, performance metric values of “perturbed models” were consistently worse than those of the “optimal model” (E avg -for-benefit≥0.070 versus 0.001, E 90 -for-benefit≥0.115 versus 0.003, cross-entropy-for-benefit≥0.757 versus 0.733, Brier-for-benefit≥0.215 versus 0.212). Calibration, discriminative ability, and overall performance of three different models were similar in the case study. The proposed metrics were implemented in a publicly available R-package “HTEPredictionMetrics”. Conclusion The proposed metrics are useful to assess the calibration and overall performance of models predicting treatment effect.
BibTeX
@misc{maas2022performance,
  author = {Carolien C. H. M. Maas and David M. Kent and Michael C. Hughes and Rommert Dekker and Hester F. Lingsma and David van Klaveren},
  title = {{Performance metrics for models designed to predict treatment effect}},
  year = {2022},
  doi = {10.1101/2022.06.14.22276387},
  howpublished = {medRxiv},
}
Figure