Model updating after interventions paradoxically introduces bias

James Liley, Samuel R. Emerson, Bilal A. Mateen, Catalina A. Vallejos, Louis J. M. Aslett, Sebastian J. Vollmer

Abstract: Machine learning is increasingly being used to generate prediction models for use in a number of real-world settings, from credit risk assessment to clinical decision support. Recent discussions have highlighted potential problems in the updating of a predictive score for a binary outcome when an existing predictive score forms part of the standard workflow, driving interventions. In this setting, the existing score induces an additional causative pathway which leads to miscalibration when the original score is replaced. We propose a general causal framework to describe and address this problem, and demonstrate an equivalent formulation as a partially observed Markov decision process. We use this model to demonstrate the impact of such `naive updating' when performed repeatedly. Namely, we show that successive predictive scores may converge to a point where they predict their own effect, or may eventually tend toward a stable oscillation between two values, and we argue that neither outcome is desirable. Furthermore, we demonstrate that even if model-fitting procedures improve, actual performance may worsen. We complement these findings with a discussion of several potential routes to overcome these issues.

Introduction

In high-stakes domains like healthcare and finance, machine learning models are increasingly deployed not just to predict outcomes, but to guide interventions. For instance, a clinical model predicting high cardiovascular risk may trigger preventative treatment. While beneficial for the patient, this intervention alters the relationship between predictors and outcomes in the resulting data. This paper explores the hazards of "naive model replacement"—the common practice of updating a model using data generated while a previous version was active—and demonstrates how this creates a feedback loop that introduces systematic bias.

The Causal Problem of Performative Prediction

The core issue is that the model becomes an active component of the system it aims to describe. In a standard observational setting, we observe covariates $X$ and outcomes $Y$ to fit an initial model. However, once that model is used to drive interventions, it creates a new causal pathway that "contaminates" future data.

Causal diagrams illustrating the impact of model-driven interventions on data generation. (A) Initial model fitting. (B) Model-driven interventions influencing subsequent data.

When we update the model using this contaminated data, the model may misinterpret the lack of outcomes (due to successful interventions) as a decrease in underlying risk. This "victim of success" effect can lead to models that underestimate risk precisely where they are most effective, potentially leading to the withdrawal of necessary interventions. This phenomenon is a specific hazard of performative prediction.

Recursive Dynamics and the Better-is-Worse Paradox

The authors formalize this process across multiple "epochs" or update cycles. By modeling the update as a recursive function and applying the Banach fixed-point theorem, they identify two primary failure modes:

  1. Stable Oscillations: If the effect of an intervention is strong enough, the model may never settle. It predicts high risk in one epoch, triggering interventions that lower observed risk, leading the next model to predict low risk and stop interventions, which causes risk to spike again.
  2. Self-Predicting Convergence: Even if the model converges, it may not converge to the "truth." Instead, it learns to predict the outcome after its own intervention, which is often not what stakeholders intended to measure.
Timeline of model updating across successive epochs, showing how previous models influence future data through interventions.

As shown in the timeline above, the model $\rho_0$ from Epoch 0 becomes an input for the data generation process in Epoch 1. This recursive dependency means that even if model-fitting algorithms improve (e.g., using more complex architectures), the actual real-world performance may worsen because the underlying intervention bias remains unaddressed—a phenomenon termed the "Better-is-Worse Paradox."

Proposed Solutions: Successive Adjuvancy

To mitigate these issues, the authors propose an alternative strategy called "Successive Adjuvancy." Instead of replacing the old model, the new model acts as an "adjuvant" (an add-on) that captures residual risk. This approach is analogous to boosting in gradient descent, where each subsequent model focuses on the errors or risks that the current system is failing to catch. They prove that under certain conditions, this framework leads to a more stable "equivocal risk" state, where the system converges to a point where everyone is brought to the same risk level, facilitating more stable resource allocation.

Conclusion

Naive model updating in the presence of interventions is fundamentally flawed. By framing the problem through causal diagrams and Partially Observed Markov Decision Processes (POMDPs), this research highlights the necessity of accounting for a model's own influence on its training data. While solutions like Successive Adjuvancy or maintaining "hold-out" sets exist, they come with significant ethical and technical hurdles that must be addressed as ML continues to integrate into decision-making workflows.

Summary updated

Abstract

Machine learning is increasingly being used to generate prediction models for use in a number of real-world settings, from credit risk assessment to clinical decision support. Recent discussions have highlighted potential problems in the updating of a predictive score for a binary outcome when an existing predictive score forms part of the standard workflow, driving interventions. In this setting, the existing score induces an additional causative pathway which leads to miscalibration when the original score is replaced. We propose a general causal framework to describe and address this problem, and demonstrate an equivalent formulation as a partially observed Markov decision process. We use this model to demonstrate the impact of such `naive updating' when performed repeatedly. Namely, we show that successive predictive scores may converge to a point where they predict their own effect, or may eventually tend toward a stable oscillation between two values, and we argue that neither outcome is desirable. Furthermore, we demonstrate that even if model-fitting procedures improve, actual performance may worsen. We complement these findings with a discussion of several potential routes to overcome these issues.
BibTeX
@inproceedings{liley2021model,
  author = {James Liley and Samuel R. Emerson and Bilal A. Mateen and Catalina A. Vallejos and Louis J. M. Aslett and Sebastian J. Vollmer},
  title = {{Model updating after interventions paradoxically introduces bias}},
  booktitle = {International Conference on Artificial Intelligence and Statistics},
  year = {2021},
  url = {https://openreview.net/forum?id=10lQv71IDRD},
}

Loading related content...

Figure