Abstract: Machine learning is increasingly being used to generate prediction models for use in a number of real-world settings, from credit risk assessment to clinical decision support. Recent discussions have highlighted potential problems in the updating of a predictive score for a binary outcome when an existing predictive score forms part of the standard workflow, driving interventions. In this setting, the existing score induces an additional causative pathway which leads to miscalibration when the original score is replaced. We propose a general causal framework to describe and address this problem, and demonstrate an equivalent formulation as a partially observed Markov decision process. We use this model to demonstrate the impact of such ‘naive updating’ when performed repeatedly. Namely, we show that successive predictive scores may converge to a point where they predict their own effect, or may eventually tend toward a stable oscillation between two values, and we argue that neither outcome is desirable. Furthermore, we demonstrate that even if model-fitting procedures improve, actual performance may worsen. We complement these findings with a discussion of several potential routes to overcome these issues.
1. Introduction
In many high-stakes domains, such as healthcare, machine learning models are deployed to predict negative outcomes so that preventative interventions can be staged. However, a fundamental paradox arises during the model update cycle: the data collected to train a new version of the model is inherently influenced by the interventions triggered by the previous version. This "naive model replacement" can lead to a phenomenon where models become "victims of their own success," eventually underestimating the very risks they were designed to detect because the interventions successfully prevented the predicted outcomes.
2. Causal Framework and Recursive Dynamics
To analyze this problem, the authors partition covariates into three categories: Fixed ($X^s$), Actionable ($X^a$, variables modified by intervention), and Latent ($X^l$, unobserved). The intervention is modeled as a function $g$ that modifies the environment based on the current model's output.
The update cycle is treated as a mathematical recursion: $\rho_{e+1} = h(\rho_e)$, where $\rho$ is the model at epoch $e$. This recursive formulation allows for the application of control theory, specifically framing the lifecycle as a Partially Observed Markov Decision Process (POMDP). Here, the "action" is not just the clinical intervention, but the choice of the next model itself.
3. Stability and the Paradox of Better Fitting
The core theoretical contribution, Theorem 1, provides conditions under which a sequence of model updates will either converge to a fixed point or enter a stable oscillation. If the derivative of the update function $h'(z) \le -1$, the risk scores will never settle, instead flip-flopping between overestimating and underestimating risk across epochs.
A counterintuitive finding is the "Paradox of Better Fitting": even as model-fitting procedures become statistically more accurate, actual real-world performance (minimizing the bad outcome $Y$) can worsen. When a model converges, it often begins to predict the post-intervention probability. In this state, the model is no longer predicting inherent patient risk, but rather the effectiveness of the intervention triggered by its own previous output.
4. Breaking the Feedback Loop
The authors identify three primary strategies to mitigate these biases:
- Complex Causal Modeling: Explicitly modeling the intervention process, though this requires access to often-unobserved data.
- Hold-out Sets: Maintaining a control group of subjects who do not receive model-driven interventions, allowing the system to observe the "natural" risk distribution.
- Control Interventions: Decoupling the update cycles of the risk model and the intervention policy to prevent direct feedback.
While these methods can stabilize the system, they present significant ethical and regulatory challenges, particularly in clinical settings where withholding a potentially beneficial tool from a subset of patients is controversial.
Summary updated