Right Diagnosis, Unsafe Route: Auditing Clinical AI Decision Trajectories and Corrective Oversight for Validity and Potential Harm

Guanqiao Chen, Wei Jiang, Shu Zhang, Qian Lin, Guanxu Yu, Siran Li, Chunhui Wang, Tianyu ShiarXiv 2026 · arXiv

In the world of clinical AI, we often obsess over the "final answer." Did the model correctly identify the myocardial infarction? If yes, we give it a gold star. But in medicine, the path to that diagnosis matters just as much as the diagnosis itself. A model that eventually gets the right answer but recommends discharging a high-risk patient on day one is not a helpful tool—it’s a liability.

A new paper titled "Right Diagnosis, Unsafe Route" exposes this "accountability gap" by auditing the entire decision-making trajectory of clinical AI systems.

TL;DR

Clinical AI systems frequently reach correct final diagnoses through intermediate steps that are clinically unacceptable or potentially harmful. Furthermore, while providing structured human feedback can fix an initial error, it often leads the AI to introduce entirely new risks downstream as the clinical case evolves.

What Problem Does This Paper Address?

Most AI evaluations are "static": they provide a complete case vignette and ask for a diagnosis. Even "stepwise" benchmarks, which release information over time, usually focus on diagnostic accuracy or the efficiency of information gathering.

The authors argue that these metrics ignore patient safety along the trajectory. A clinical decision-support system (CDSS) makes a series of recommendations—tests to order, treatments to start, whether to escalate care. If an AI recommends omitting a time-critical antiplatelet drug on day one, it has failed, regardless of whether it correctly identifies the heart attack on day three.

The paper also tackles the "illusion of oversight." Regulations often mandate "human-in-the-loop" systems, but if a human simply rejects an AI's bad idea without providing context, does the AI actually recover safely?

Methods

The researchers developed a framework called ClinTraj to conduct two distinct types of audits:

  1. Trajectory Audit: This "replays" real patient admissions checkpoint-by-checkpoint. At each stage, the AI sees only the evidence available to a doctor at that specific time. An LLM-based "judge" (calibrated against a physician reference standard) rates every decision for Clinical Validity ($V$) and Potential Harm ($H$).
  2. Oversight Audit: When an AI makes its first "actionable error," the researchers branch the simulation into four experimental arms to test different types of feedback:
  3. C0: No correction (Control).
  4. C1: Bare rejection ("Reconsider").
  5. C2: Structured correction (Problem + Corrected Action + Rationale).
  6. C3: The same content as C2, but in plain prose.
Fig. 1. The trajectory and oversight audits at a glance. ❶A case record becomes a trajectory of decision checkpoints 𝑇0, . . . ,𝑇end; objective evidence is released in the order clinicians could have known it, clinicians’ own judgements never reach a system, and the documented final diagnosis anchors the audit. ❷At every checkpoint a system decides with only the evidence released so far. In the trajectory audit, an LLM judge that knows the final diagnosis but not the system scores every decision for potential harm and clinical validity on the evidence available at its checkpoint, and clinicians score a stratified sample to calibrate its rates; in the oversight audit, each trajectory is branched at its first actionable error. ❸The two findings, shown for the hospital cohort with the other sources below; Figure 9 shows every system on every source.

To ensure the LLM judge wasn't hallucinating safety standards, the authors used a physician reference standard: ten clinicians independently scored a sample of trajectories, with senior physicians adjudicating disagreements. This allowed them to "physician-calibrate" the AI judge's findings.

Datasets

The study evaluated seven AI systems (including GPT-4 variants, MedGemma, and specialized clinical agents) across three diverse sources:

  • Hospital Cohort: 96 cardiology admissions from a tertiary teaching hospital in China.
  • MIMIC-IV: 96 emergency cardiology admissions from a US-based center.
  • ASCENT: 108 multi-specialty vignettes from a public diagnostic reasoning benchmark.

Key Insights and Ideas

  • The "Correct Diagnosis" Mask: A correct final diagnosis is a poor proxy for safety. In the hospital cohort, over half of the trajectories that ended in a correct diagnosis passed through at least one "potentially unsafe" decision.
  • Omission is the Primary Failure: The most common type of harm wasn't the AI recommending a "poisonous" treatment, but rather critical omissions (42% of failures)—failing to order a time-sensitive test or treatment.
  • The "Stale" Correction Problem: This is perhaps the most fascinating finding. When a human provides a structured correction, the AI often adopts it. However, as new evidence arrives at the next checkpoint, the AI may cling to that "corrected" framing even if it's no longer appropriate, leading to new harms.
  • Judge Divergence: LLM judges and human clinicians agree reasonably well on whether a diagnosis is "acceptable," but they diverge sharply on potential harm. LLMs tend to be more "alarmist," rating many more decisions as potentially unsafe than human physicians do.
The trajectory audit process, showing how decisions are decomposed and scored.

Key Results

  • Safety Gaps: Among hospital trajectories reaching an acceptable final diagnosis, 55.2% passed through a decision with moderate or severe potential harm. On the MIMIC-IV dataset, this rose to a staggering 76.2%.
  • Oversight Efficacy: Structured feedback (C2) resolved 73.6% of first errors, compared to only 24.8% for a simple "reject and reconsider" (C1).
  • The Recovery Trap: Repairing an error does not guarantee a safe trajectory. 30.2% of trajectories that were "corrected" by structured feedback introduced a new potentially unsafe element within the next two checkpoints.
Fig. 2. The trajectory audit. ❶A case record is replayed as checkpoints from admission (𝑇0) to the final checkpoint (𝑇end): objective evidence becomes visible once it was knowable, clinicians’ judgements never, and the documented final diagnosis is the audit’s reference. ❷At checkpoint 𝑡a system decides with only the evidence released by 𝑡and its own earlier decisions. ❸A system-blind LLM judge rates every atom of the decision for clinical validity (𝑉: 2 acceptable, 1 suboptimal, 0 unacceptable) and potential harm (𝐻: 0 none to 3 severe); the decision takes the lowest 𝑉and the highest 𝐻, and 𝐻≥2 marks it potentially unsafe. ❹A trajectory with an acceptable final diagnosis (𝐸= 1) that passes through a potentially unsafe (𝑈= 1) or unacceptable (𝐼= 1) intermediate decision counts toward cutr or citr. ❺Clinicians re-rate a stratified sample of trajectories, and the sampling weights turn their labels into the physician-calibrated rates of Section 4. Example values are illustrative.

Limitations and Future Work

The authors acknowledge that their audit is limited to cardiology and a specific set of datasets. There is also the inherent risk of hindsight bias; even though the judge is instructed to only use evidence available at the checkpoint, knowing the "ground truth" final diagnosis might subtly influence the scoring.

Future work needs to move toward dynamic oversight. Instead of one-off corrections, we need systems that monitor the entire trajectory and ensure that as the clinical "facts on the ground" change, the AI’s reasoning—and the human’s oversight—evolves with them.

Conclusion: If we are to deploy AI in high-stakes clinical settings, we must stop grading it on the final exam alone. We need to audit the "unsafe routes" and ensure that our methods of human oversight aren't just fixing one leak while the ship continues to sink.

Summary updated

Abstract

A clinical AI system can reach a correct final diagnosis through intermediate decisions that could have harmed the patient, and can repair an error while introducing new risks. Most evaluations score final diagnoses; even stepwise benchmarks, which release a case's evidence over time, score whether systems reach the right diagnosis or gather the right information, not whether the decisions a patient passes through are safe. We make patient safety the target of stepwise evaluation. A trajectory audit replays real admissions checkpoint by checkpoint, lets each system decide with only the evidence released so far, and rates every test, treatment and disposition for its potential harm to the patient and, separately, for clinical validity, against the documented final diagnosis and physician-written disease rubrics yet on the evidence available when the decision was made. An LLM judge scores every decision, and a physician reference standard (triple annotation and senior adjudication of a stratified sample) calibrates the rates. An oversight audit branches each trajectory at its first actionable error and measures, on the same branches, whether corrective feedback repairs the error and whether it introduces new harm. Across 300 cases from a hospital cardiology cohort, MIMIC-IV and the public ASCENT benchmark, with up to seven AI systems per source, 55.2% of hospital trajectories reaching an acceptable final diagnosis passed through a decision with moderate or severe potential harm and 24.8% through a clinically unacceptable one; the harm rates were 76.2% on MIMIC-IV and 39.6% on ASCENT. Structured corrections resolved 73.6% of first errors versus 24.8% after bare rejection, yet new potentially unsafe decisions followed 30.2% of corrections versus 19.7% of rejections. A correct final diagnosis does not certify a safe trajectory, and a successful correction does not certify a safe recovery.
BibTeX
@misc{chen2026right,
  author = {Guanqiao Chen and Wei Jiang and Shu Zhang and Qian Lin and Guanxu Yu and Siran Li and Chunhui Wang and Tianyu Shi},
  title = {{Right Diagnosis, Unsafe Route: Auditing Clinical AI Decision Trajectories and Corrective Oversight for Validity and Potential Harm}},
  year = {2026},
  eprint = {2611.22612},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.22612},
}
Figure