When a Clinician Accepts the Machine's Mistake: Measuring and Stopping Error Cascades in Multi-Turn Clinical AI

Studies of how clinicians use AI advice almost always test one decision at a time. A physician sees one recommended drug dose, or a pathologist sees one estimate, accepts or rejects it, and the case ends. Real clinical work with a conversational assistant does not end there. A clinician might ask for a patient's lactate trend, then a severity score built on that trend, then a dosing suggestion built on the score. If the clinician waves through a subtly wrong answer early on, that answer becomes part of the conversation history. The language model then treats it as settled fact for every later step. This happens through what the error-propagation literature calls snowballing: models prefer staying consistent with their own earlier outputs over checking them against evidence. Nobody has measured what happens when this model-side tendency meets the human-side tendency to defer under time pressure. In particular, no one has checked whether the high rejection rate clinicians show for single unsafe recommendations survives once an error has already been accepted and built upon.

We propose to build a physician-verified benchmark of multi-turn ICU record sessions. Each session contains one deliberately planted error, and every later turn carries a label saying whether the planted error contaminated it. We use the benchmark for three things. The first is to measure cascades in models alone. The second is a time-pressured clinician study that tests whether human vigilance decays along a session and collapses after an earlier acceptance. The third is to test a cheap inference-time check: an independent model recomputes each key value from the raw record without seeing the conversation, and the system raises a flag directly on the recommendation when the two values disagree.

Core Claim

The claim is that clinical human-AI safety is a property of trajectories, not single decisions, and that it can be measured in a controlled way. Doing so needs one ingredient no existing resource has. For each turn after a planted error, we need a verified answer to the question "would this output differ if the earlier error had not been made?" Once those labels exist, we can separate three things. One is errors the model inherits from its own history. Another is errors that human endorsement locks in. The third is errors a clinician would catch if the warning were placed where they actually look.

What Prior Work Establishes and Leaves Open

The ecological validity literature motivates the human side of the project. Festor et al. put 38 ICU physicians in a physical simulation suite. The physicians rejected 92% of unsafe AI doses, but their gaze went to the recommendation screen and almost never to the explanation panel or back to the chart. Each of their six scenarios was a single independent decision. Rosbach et al. found a 7% automation-bias rate among 28 pathologists. They also found that a 10-second limit acts as a severity multiplier: under time pressure, clinicians were not fooled more often, but when they were fooled they moved further toward the wrong advice. That study, too, used isolated image patches. Collab-CXR showed that clinical history helps radiologists override wrong AI reads, again on static cases. Nagendran et al. constrains our design: what clinicians say they value in explanations does not predict their actual choices. For that reason we put our intervention on the recommendation itself rather than in an explanation pane.

The error-propagation literature supplies the model side and part of the method. Zhong et al. built MMHalSnowball by having ChatGPT inject fake prior turns. Open-source vision-language models lost at least 31% accuracy as a result. Their remedy, residual visual decoding, compares predictions made with and without the conversation history. Our recompute check borrows this "history-free reference" idea and moves it from images to the patient record. Their benchmark is general-domain and has no expert validation. Wang et al. showed that agents reuse their first-turn tool 76–89% of the time, and that prompting "never reuse" cuts this by only 5–10%. That result constrains us: we do not rely on instructions to fix the problem. LLM-as-RNN rewrites a natural-language memory at each step. It self-corrects prior errors in 54.8% of MIMIC-IV next-diagnosis cases, but it works purely offline and the errors it corrects are its own. ACR restructures context with operators such as fact rectification, but it was tested only on general question answering.

The human-in-the-loop synthesis literature supplies the construction method. EHRSQL turned 1,742 real questions from hospital staff into 230 templates over MIMIC-III and eICU. 93.2% of those templates involve time constraints, which makes them ideal places to plant subtle errors such as a wrong time window. Ye et al. found that roughly 27% of MedCalc-Bench labels were clinically wrong. They repaired them in three phases: an automated audit, blinded independent recomputation from the raw notes, and physician adjudication of only the disputed cases. We reuse this pipeline to label contamination. Both resources are single-turn, which is the gap we fill.

Research Program

Sessions. Each session follows one MIMIC-III or eICU ICU stay through a chain of five to eight turns. The chain starts with record queries drawn from EHRSQL templates, moves to clinical scores in the style of MedCalc-Bench, and ends with a management suggestion. A generator model drafts the conversational wording.

Planted errors. Into each session we plant exactly one error at a chosen turn. The error types are a shifted time window, a wrong unit, a dropped exclusion, or a misread value. The planted error is always plausible on its face.

Contamination labels. For every later turn we compute two answers programmatically: the answer from the clean chain and the answer from the corrupted chain. Where they differ, the turn is labeled contaminated. Where a step is not computable from the record, an independent model recomputes it blind, as in Ye et al.'s second phase. Physicians adjudicate only the turns where the program, the recomputation, and the generator disagree. This keeps scarce physician time on the cases that need it. Ye et al. adjudicated 50 cases; we budget for a few hundred disputed turns.

Measurements. We measure three things.

The first is model-only cascades, comparing two conditions. In one, the model sees a gold history. In the other, the planted error appears in the history either as the model's own output or as an output the clinician has explicitly endorsed. That second contrast tests a hypothesis no prior work has examined: that human endorsement makes models even less willing to revisit an error.

The second is a web-based clinician study with time-pressure and untimed arms, following Rosbach et al. We vary the turn at which the error is planted, and whether the clinician accepted an earlier error, within each participant. The study records how often clinicians reject errors, their weight of advice (how far they move toward the AI's suggestion, estimated with mixed-effects models), and their click and dwell behavior on the raw-record panel, in the style of Collab-CXR. We do not attempt a physical simulation; we state this limitation openly.

The third is the recompute check. A model that does not see the conversation recomputes the key value directly from the record. When its value differs from the conversation's value, an inline discrepancy marker appears on the recommendation itself.

Baselines

Do nothing. The naive reference is the standard append-only assistant: plain concatenation of the full history, run on the same base model. It defines the cascade rate that every mitigation must reduce.

Prompting. The second baseline is a "re-verify every prior value" system prompt, tuned on a development split with the same number of prompt variants we allow ourselves. Wang et al. report that this kind of instruction reduced anchoring by only 5–10% on TBMT. Beating it means a substantially larger drop in contamination on our benchmark.

Mutable memory (strongest existing method). The strongest competitor is LLM-as-RNN's mutable memory. It is the only cascade method already shown to work on clinical records: a 10.8% relative gain over full-history accumulation on MIMIC-IV, with 54.8% self-correction of prior errors. We run it with the same base model and the same token budget per turn as our check. We count tokens rather than steps, because its generate-reflect-update loop uses more inference compute. ACR is run the same way as a second strong competitor.

No reported result was found on Lacuna for any method on multi-turn clinical sessions with planted errors. Every number on our benchmark will therefore be new, and the comparison is purely relative. Beating mutable memory means fewer contaminated downstream turns at equal tokens. It also means not falsely flagging clean turns: we require the false-flag rate to stay within a few points of the do-nothing system's error rate on clean sessions.

Ablation of the key ingredient. The ablation runs the identical recompute check but lets the checker see the conversation history. If this version performs as well as the blind version, then blindness to the conversation is not what does the work.

Human reference. The human-side reference is the 92% single-turn rejection rate from Festor et al. Our study tests whether that rate holds at later turns and after an earlier acceptance.

Why not just…?

Why not just generate the benchmark fully automatically, as MMHalSnowball did? This is a real threat to the physician phase. Ye et al. found shared blind spots when the generating model and the auditing model come from similar model families, and found that original labels agreed with physicians on only 20% of contested cases. We will test the threat directly. On the adjudicated subset, we will measure how often automated-only contamination labels agree with physicians. If agreement is high, we will say the physician phase is unnecessary for this task family, and the pipeline will become cheaper.

Why not just evaluate models in silico with self-generated histories? That captures half the phenomenon and is the first measurement in our program. It cannot show whether clinicians catch inherited errors, or whether human acceptance changes model behavior. If the clinician study finds that rejection rates are flat across turns and do not depend on earlier acceptance, single-turn human studies plus in-silico cascade tests would be enough. We would report that as a clear negative result.

Why not just deploy mutable memory or context refactoring? These methods correct errors the model judges to be wrong. An error a clinician has explicitly accepted may look validated and survive rewriting. The endorsed-versus-self-generated contrast tests this. If mutable memory corrects endorsed errors as well as self-generated ones, at lower cost than our check, we will recommend it instead.

Why not just show better explanations? Festor et al.'s eye-tracking shows that clinicians under load do not look at explanation panels. That is why our flag sits on the recommendation. A comparison arm in the clinician study shows the same discrepancy in a side explanation panel instead.

Decisive Checks

Three results would make the project's case. The first is that contamination in model-only runs exceeds the gold-history rate, and exceeds it more when the planted error has been endorsed by a clinician. The second is that clinician rejection of planted errors falls with turn position or after an earlier acceptance, with the drop larger under time pressure. That would extend Rosbach et al.'s severity-multiplier effect to trajectories. The third is that the blind recompute flag reduces both model contamination and human acceptance more than the prompting baseline, and more than mutable memory at equal tokens, without raising false flags on clean sessions.

If none of these hold, single-decision evaluation is vindicated. The verified benchmark would remain a reusable resource either way.