Performative Feedback Loops in Clinical Machine Learning

26 concepts18 papers1 proposal

When we train a machine learning model to predict the weather, our prediction does not change whether or not it rains. The model is a passive observer of a stationary physical process.

In healthcare, this passive observer assumption completely breaks down. When a clinical algorithm predicts that a patient is at high risk of dying, clinicians do not sit idly by; they act on that prediction. They might initiate intensive care, alter drug dosages, or, in tragic circumstances, decide to withdraw life-sustaining therapy.

This interaction creates a performative feedback loop: the act of making a prediction alters the very outcome the model is trying to predict. When these post-prediction interventions are recorded back into electronic health records, they corrupt future training data, leading to severe algorithmic biases that can cause models to dangerously degrade over time.

The Two Flavors of Performative Bias

Performative loops manifest in two distinct ways depending on whether the clinical intervention aims to prevent the predicted outcome or concede to it.

1. The Self-Defeating Prophecy (The Success Paradox)

Consider a model designed to predict a patient's risk of a heart attack or a cancer relapse. If the model flags a patient as high-risk, clinicians will immediately deploy preemptive treatments to mitigate that risk. As highlighted by Shuang Fan et al. in the context of leukemia relapse, these intermediate interventions mask the natural progression of the disease.

If the intervention is successful, the patient does not suffer the adverse outcome. When this data point is saved, the model's history shows: Prediction: High Risk $\rightarrow$ Outcome: Healthy.

To a standard evaluation metric, this looks like a false positive. If the model is subsequently retrained on this data, a paradox occurs. As James Liley et al. demonstrate, the model begins to mistake the post-intervention safety for a lack of underlying risk. It learns that patients with these severe symptoms end up fine, causing it to stop flagging critical risks entirely. The model becomes a victim of its own success, systematically underestimating the hazards it was designed to prevent.

2. The Self-Fulfilling Prophecy (The Death Loop)

The opposite occurs when a model predicts a catastrophic outcome, such as a comatose patient failing to achieve neurological recovery. If clinicians trust a pessimistic prediction, they may decide on the withdrawal of life-sustaining measures (WLSM).

This clinical action guarantees the predicted outcome (death), regardless of whether the patient actually had the biological potential to recover. As Jieshi Chen et al. note, this intervention ensures the dataset is heavily biased toward confirming the model's initial intuition. Standard performance metrics like the Area Under the ROC Curve (AUC) will deceptively appear stable or even perfect, masking the fact that the model is simply predicting the downstream effects of its own influence.

[Model Predicts High Risk/Mortality]
               │
               ▼
   [Clinician Alters Treatment]
         /           \
        /             \
(Preventive Care)     (Withdrawal of Care)
      │                       │
      ▼                       ▼
[Outcome Prevented]     [Outcome Guaranteed]
      │                       │
      ▼                       ▼
Self-Defeating          Self-Fulfilling
   Prophecy                Prophecy
(Retraining drops       (Deceptive high
 risk sensitivity)       accuracy metrics)

The High Stakes of Neuroprognosis

The self-fulfilling prophecy is particularly acute in critical care settings. Research by Ernest van Veen et al. shows that in patients with severe traumatic brain injury, the vast majority of ICU deaths occur following a decision to withdraw life-sustaining measures, often within the first 72 hours.

When prognostic models are built using historical data, they risk cementing outdated clinical intuitions or institutional biases into automated code. As Michael Müller et al. argue, validating a prognostic tool becomes highly confounded when the tool's primary inputs are the exact same features clinicians use to make life-ending decisions. This loop is not unique to machine learning; even traditional clinical scores suffer from it. For instance, Christian Jung et al. observed that when clinicians are explicitly aware of a patient's frailty score, that knowledge alone can bias treatment choices, artificially worsening outcomes for frail patients and creating a false validation of the score's inherent biological predictive power.

To break this loop in coma recovery modeling, the George B. Moody PhysioNet Challenge 2023 motivated the development of open-source algorithms designed to identify poor neurological outcomes with extreme specificity, minimizing the risk of false negatives that could lead to premature withdrawal of care. Similarly, Felix Krones et al. emphasize that developing objective, multimodal deep learning models is essential to bypass the subjective over-optimism or pessimism that Anand Avati et al. note often leads to treatment inertia or delayed palliative care.

Why Standard Model Updates Fail

When a model's performance begins to drift, the standard engineering response is to retrain the model on the most recent data. However, in the presence of performative feedback loops, this practice is actively harmful.

George Alexandru Adam et al. mapped out how this retraining loop causes severe model degradation. If a clinician trusts an incorrect prediction and alters treatment, that action alters the outcome data. When the model is retrained on this corrupted data, it reinforces its own initial error.

Jean Feng et al. describe this as a paradox where successful clinical utility manifests as statistical error. Because the model's deployment alters the data distribution, treating the post-deployment environment as a standard stationary dataset leads to "algorithmic confounding."

To safely deploy and update clinical AI, we must transition from purely correlational models to frameworks that account for causal domain shifts.

Causal Domain Monitoring

Instead of treating model updating as a standard supervised learning problem, the authors Philip A. Boeken et al. frame the deployment of decision support systems as a causal domain shift. By modeling the clinician's intervention as an explicit causal node, researchers can mathematically adjust for performative bias, allowing them to estimate the true underlying patient risk even when randomized controlled trials are unethical.

Jean Feng et al. have formalized this into post-market monitoring frameworks for machine learning-based medical devices. Their framework uses causal diagrams to decouple the model's intrinsic accuracy from the behavioral responses of the physicians, ensuring that a preventive "success" is not mistakenly flagged as an algorithmic failure.

Passive Observation:   [Patient Features] ───────────────────────────────► [Outcome]

Performative Loop:     [Patient Features] ───► [Model Prediction] 
                               │                      │
                               ▼                      ▼
                       [True Baseline Risk] ──► [Clinical Action] ──────► [Observed Outcome]

Empirical Disentanglement

On the clinical side, we can mitigate self-fulfilling prophecies through rigorous validation design. Benjamin Rohaut et al. demonstrated that performative bias can be empirically isolated by performing sub-analyses that explicitly exclude cases where predictions directly influenced the final outcome—such as validating neuroprognostic models strictly on a cohort of patients who received continued, uninterrupted life-sustaining therapy. This ensures that the validated accuracy reflects the true underlying state of the patient's physiology rather than the downstream consequences of the model's own output.

Key Foundational Work

Predicting neurological recovery with Canonical Autocorrelation Embeddings (2019) — contributed 1 concept elements to this direction.
Survival and consciousness recovery are better in the minimally conscious state than in the vegetative state (2017) — contributed 1 concept elements to this direction.
Improving Palliative Care with Deep Learning (2017) — contributed 1 concept elements to this direction.
History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 16 papers
  2. Version 1 · 2026-10-04 · 19 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_prognostic_self_fulfilling_prophecy_1b8d8023,
  title = {prognostic self-fulfilling prophecy},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/prognostic-self-fulfilling-prophecy/rdr_1b8d80239fd0469d9cb5a153bd45b741},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 2, frozen text):

@misc{lacuna_prognostic_self_fulfilling_prophecy_1b8d8023_v2,
  title = {prognostic self-fulfilling prophecy},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/prognostic-self-fulfilling-prophecy/rdr_1b8d80239fd0469d9cb5a153bd45b741/v/2},
  note = {Version 2, frozen 2026-10-04. Accessed 2026-11-21},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake