Where Diagnosis Breaks on Lookalike Syndromes: Tiered Minimal-Pair Tests for CRS versus Sepsis

After CAR-T therapy, a cancer treatment that reprograms a patient's own T cells to attack tumor cells, patients often develop cytokine release syndrome (CRS). CRS is an immune overreaction that causes high fever, low blood pressure and organ dysfunction. Bacterial sepsis produces the same bedside picture, and the two peak in the same first weeks after infusion. The treatments point in opposite directions. Immunosuppressants such as tocilizumab or steroids can turn occult sepsis into fatal shock, while antibiotics alone leave CRS unchecked. We propose an evaluation that asks whether a diagnostic model, whether a clinical LLM or a time-series classifier, actually reasons through this lookalike problem. Each patient case becomes three layers of questions. The first asks whether the model reads the abnormal vitals and labs correctly. The second asks whether it tracks how they change over hours. The third asks whether it picks the right diagnosis. We pair every case with a "minimal twin" that changes only the evidence separating CRS from sepsis, such as culture results, procalcitonin trend, or timing relative to infusion.

The core claim is that one accuracy number cannot tell us whether a model separates lookalike syndromes for the right reasons. Two ingredients together can. Tiered questions locate where the reasoning fails. Paired counterfactual cases test whether the final call depends on the discriminating evidence or on shortcuts shared by both syndromes. Existing hierarchical diagnosis benchmarks organize diseases by billing-code ancestry, and billing codes place CRS and sepsis in unrelated branches. They therefore cannot express the question that matters most in the CAR-T intensive care unit (ICU).

What Exists and What Is Missing

Hierarchical evaluation is well developed outside this problem. Benchmarks for video understanding (MVU-Eval), tool use (ToolBeHonest) and anesthesiology (AnesSuite) split tasks into perception, relational interpretation and synthesis tiers. AnesSuite found that scaling from 7B to 72B parameters helps factual retrieval far more than multi-system crisis reasoning. That result motivates us: it suggests failures concentrate in the upper tiers, but it was measured on textbook vignettes rather than patient trajectories.

The closest work is H-DDx, which scores LLM differential diagnoses on the ICD-10 tree. Its metric gives partial credit when a model reaches the right chapter but misses the exact code, and it reports accuracy near 60% at chapter level collapsing to 10–20% at subcategory level. We borrow its idea of partial credit for clinically sensible near-misses. We reject its distance measure. Under ICD ancestry, listing CRS for a septic CAR-T patient counts as badly as listing an unrelated psychiatric disorder, even though weighing both is standard practice (Valade et al.). H-DDx, DDXPlus and DDxT also all work on static synthetic outpatient symptom lists, with no time dimension. DDxT reaches 99.98% top-1 pathology accuracy there, which says little about ICU mimics.

On the clinical side, M2-CRS augments a 9-patient CAR-T cohort with statistics from 17 published trials and predicts CRS versus no CRS from 14-cytokine panels. It never faces sepsis as the alternative, and it assumes cytokine panels that are rarely available at the bedside. We reuse its technique of literature-informed distributions to generate CRS trajectories.

Two clinical papers motivate the regime and supply the discriminating axes. Valade et al. document an explicit "Sepsis or CRS" overlap group among 71 ICU CAR-T recipients, with 71% of bacterial infections catheter-related. An et al. show that bacterial infections cluster in the same early window as CRS.

Standard ICU benchmarks constrain our data design. YAIB shows that preprocessing choices matter more than architecture for sepsis prediction. SBRD models clinician treatment regimes in MIMIC-IV sepsis. Neither contains a CRS arm.

Core Claim

A mimicry benchmark should be organized by where the evidence for two syndromes diverges, not by disease ontology. The shared phenotype sits at the bottom, the divergent kinetics in the middle, and the discriminating rule-out evidence at the top. Every top-tier item should come with a minimal twin.

We expect current models to answer bottom-tier questions well, to degrade on kinetics, and to change their top-tier answer between twins far less often than the changed evidence warrants. That last pattern would show they rely on the shared phenotype plus priors such as "recent CAR-T means CRS."

Research Program

We build trajectories from two sources.

The sepsis arm uses real MIMIC-IV and eICU sepsis stays, extracted through the YAIB harmonized pipeline. These carry real missingness and irregular sampling.

The CRS arm cannot come from public data, since CAR-T cohorts are tiny and sit in specialist centers. We will therefore build it in two ways. First, we generate semi-synthetic trajectories. We take real ICU shock trajectories with negative cultures and re-time and re-shape the inflammatory markers toward literature-reported CRS kinetics, following the M2-CRS approach of using trial-reported medians and ranges. Second, we seek a small real CAR-T ICU cohort through a partner center, comparable to the one in Valade et al., to use strictly as a held-out test set.

Minimal twins are made by editing only discriminating variables. These include blood culture result and its timing, procalcitonin trajectory, time since infusion, and presence of a central line. Everything else stays fixed: the fever curve, blood pressure, vasopressor dose and organ-dysfunction score. Two ICU and hematology clinicians will review a random sample of twins to confirm that the edit changes the correct label and that the result looks plausible.

Each case then yields questions at three tiers. Tier 1 asks for the current values and abnormality of features. Tier 2 asks for direction and rate of change, and for ordering of events such as "did fever precede the culture draw." Tier 3 asks for a ranked differential and the immediate management choice.

The primary metric is a pair-consistency score:

$$ S_{\text{pair}} = \frac{1}{|P|}\sum_{(x,x')\in P} \mathbf{1}\big[f(x)=y \,\wedge\, f(x')=y'\big] $$

Here $P$ is the set of twin pairs, $x$ and $x'$ are a case and its twin with differing true labels $y$ and $y'$, and $f$ is the model's top-tier call. The complementary shortcut rate is the fraction of pairs where $f(x)=f(x')$.

We also report tier-conditional accuracy: top-tier accuracy restricted to cases where the model answered the lower-tier questions correctly. This measures composition failures separately from perception failures.

Finally, we report a harm-weighted error that clinicians assign in advance. It penalizes giving immunosuppression to sepsis most heavily, and replaces ICD distance with a mimicry-aware partial credit.

Baselines

The naive reference points are two simple rules. One always predicts sepsis, the safe default. The other predicts CRS whenever the case falls within 14 days of infusion. The second rule is the shortcut we suspect models learn, so any model should clearly beat it on $S_{\text{pair}}$, where by construction it scores zero.

The strongest flat competitor is a gradient-boosted tree and a GRU classifier trained directly on the four-way label (CRS, sepsis, both, neither). Both use the same trajectories and features and follow the YAIB preprocessing and Bayesian tuning protocol, with an equal tuning budget for every model. YAIB reports no sepsis AUROC in the file; its mortality baseline AUROC is 0.84 to 0.87. This baseline is the one most likely to match top-tier accuracy. The benchmark earns its keep if models with similar flat accuracy differ substantially on $S_{\text{pair}}$ or tier-conditional accuracy.

The literature-augmented baseline reimplements the M2-CRS Random Forest with soft majority voting, restricted to routinely available labs. M2-CRS reports 94.1% accuracy and AUC 0.936 on binary CRS detection from full cytokine panels, and its plain Random Forest reports 89.3% accuracy and AUC 0.881. These numbers are not comparable to our CRS-versus-sepsis task. For that task we found no reported result on Lacuna.

For LLMs, we evaluate models across the scale range used in AnesSuite, plus the model H-DDx found strongest overall, Claude-Sonnet-4. H-DDx gives no numeric score for it in the file. Every model receives identical serialized trajectories and a fixed chain-of-thought prompt, and is scored both with ICD-based partial credit and with our metrics.

Two ablations isolate the proposal's ingredients. The first removes the twins and scores only unpaired top-tier accuracy. This tests whether pairing reveals anything the flat score hides. The second removes the tier questions and scores only top-tier outputs. This tests whether the tiers add diagnostic information beyond the pairs.

Why not just…?

Why not just train a flat four-way classifier and report accuracy? For prediction alone this may work about as well, and we will say so if it does. But accuracy alone cannot tell a model that reads culture results from one that keys on infusion timing, because in real cohorts those variables are correlated. The twin experiment settles this directly. If the flat classifier has high accuracy but a high shortcut rate, the flat score was hiding a dangerous failure. If its shortcut rate is low, the flat approach is adequate for this model class, and the benchmark's value shifts to evaluating LLMs.

Why not just replace the ICD tree with a clinician-written confusion-cost matrix? This is the most serious threat to our metric design, and the harm-weighted error is essentially this idea, so we adopt it. A cost matrix scores the final answer but still cannot say whether an error came from misreading a lab, misreading a trend, or misapplying a rule-out. We will measure how much tier-conditional accuracy explains model rankings beyond the cost-matrix score. If it explains little, the tiers are not worth their construction cost.

Why not just apply H-DDx or prompt a larger clinical LLM? H-DDx's distance places CRS and sepsis at root distance, so it will mis-rank sensible differentials. We test this by correlating both metrics' model rankings with blinded clinician rankings on a subset. On scale, AnesSuite and DR.BENCH report that bigger or domain-adapted models gain little on complex reasoning. DR.BENCH reports MedQA at 20–24%. Our scale sweep checks whether this holds for kinetics and twin consistency specifically.

Decisive Checks

The first check is localization. We will report whether top-tier errors mostly occur when the lower tiers were answered correctly, which would point to a composition failure, or mostly when they were not, which would point to a perception failure. We also check whether the answer differs between trees and LLMs.

The second check is shortcuts. We will measure whether $S_{\text{pair}}$ sits well below unpaired accuracy for the strongest models, and whether the gap persists on the real CAR-T held-out cohort. If the gap disappears there, the semi-synthetic twins are introducing artifacts.

The third check is metric validity. We will test whether mimicry-aware scoring agrees with clinician rankings better than ICD-based scoring does.

The main risk is that semi-synthetic CRS trajectories are easier to detect than real ones. The real held-out cohort and the clinician review of twins are the guards against this. Any conclusion that fails to transfer to real CAR-T patients will be reported as such.