Guideline-Washing: Can Clinical Explanations Be Faked, and Can an Auditor Catch It?

Hospitals and regulators increasingly accept a clinical prediction model when its explanations "look right." In practice this means the features it ranks as important, such as lactate, blood pressure or creatinine, match what clinical guidelines say should matter. This proposal asks whether that check can be gamed. Could a model actually predict mortality from a hospital's insurance codes or a site-specific administrative token, while its explanations confidently cite guideline biomarkers? We call this guideline-washing. We propose to build such models deliberately on real intensive-care records and to design an audit that catches them. Crucially, we will evaluate that audit against an attacker who knows exactly how it works.

The core idea fits in two sentences. An explanation is a claim about what would happen to the prediction if certain inputs changed, so an auditor can test the claim by changing those inputs realistically and watching the output. If a model's real reliance is elsewhere, the gap between what the explanation claims and what the interventions show should be measurable. Hiding that gap should force the attacker to genuinely reduce its use of the shortcut, which is the outcome an auditor wants.

Why this is open

Three lines of work show that explanations can be engineered to deceive, all outside medicine. Fairwashing: the risk of rationalization fits rule-list surrogates (simple if-then explanations of a black-box model) that conceal bias, reporting 90.8% fidelity with an unfairness score of only 0.058 on Adult Income. Fairwashing explanations with off-manifold detergent shows that gradient and relevance-propagation heatmaps can be bent into arbitrary targets while accuracy changes by less than 0.2%. Its manipulation exploits input directions where no real data lives ("off-manifold"). How do I fool you? substitutes plausible correlated features for prohibited ones at 98.2% fidelity. In that study, expert trust in a biased model rose from 9.1% to 88%. These papers motivate the threat, and their attacks become our attack families. All of them target fairness claims on tabular, criminal-justice or toy image data, though. None targets the claim clinicians actually check, which is alignment with clinical guidelines.

Clinical interpretability work relies on exactly that check. It extracts SHAP rankings (a method that splits a prediction into per-feature contributions) and compares them to expert consensus, as in Assessing the Interpretability of Machine Learning Models in Early Detection of Alzheimer's Disease. It also induces clinical rule lists, as in Contextual explanation rules for neural clinical classifiers, whose UNRAVEL method reports about 99% fidelity on synthetic sepsis data. Neither work considers an adversary. MIMIC-IF benchmarks 12 attribution methods on MIMIC-IV mortality with remove-and-retrain testing (ROAR) and audits fairness, but under benign conditions. CliniCARE-Bench shows that correct outcomes hide bad process: roughly one in five correct verdicts from the most affected systems violated a prohibited shortcut criterion. That is direct evidence that output-level checks miss shortcut use, but its environment is assumed trusted.

The existing detector is Washing The Unwashable. Its FRAUD-Detect method shows that fairwashing leaves a gap in surrogate fidelity across demographic groups, and that evading it forced parity gaps above 10%. That result constrains our design in a useful way. It suggests detection should rest on a quantity the attacker cannot move without paying a price. But FRAUD-Detect treats demographic correlation as illegitimate. In medicine, Detecting shortcut learning for fair medical AI using shortcut testing shows that age and sex can be causally relevant, and that encoding a variable does not mean using it. So a demographic divergence test cannot tell washing from pathophysiology. Our audit therefore stratifies by suspected shortcut features, such as administrative, site and insurance variables, rather than by protected groups. It also tests reliance through intervention rather than through fidelity alone.

The security-validation literature supplies the methodological lens. Systems in that literature are routinely judged by fixed sanity checks rather than adaptive attack. Examples include the empirical-randomness evaluations in S-Box Design Based on 2D Multiple Collapse Chaotic Map, and MedGazeShift, whose authors note that defenses must be tested across a wider range of threats. Explanation audits have the same blind spot, so our evaluation always includes an attacker who knows the audit.

Core Claim

A query-access auditor can detect guideline-washing on longitudinal intensive-care records by comparing each explanation's claimed attribution against the effect of realistic, data-consistent interventions. For an evaluation case $x$, model $f$, and feature set $S$, let $a_S(x)$ be the share of attribution the explanation assigns to $S$. Define the measured interventional share as

$$ i_S(x) = \frac{\mathbb{E}_{\tilde{x}_S \sim p(x_S \mid x_{-S})}\,|f(x) - f(\tilde{x}_S, x_{-S})|}{\sum_{T} \mathbb{E}\,|f(x) - f(\tilde{x}_T, x_{-T})|}, $$

where $\tilde{x}_S$ redraws the features in $S$ from an auditor-trained conditional model given the remaining features $x_{-S}$, and $T$ ranges over a fixed partition of feature groups. The audit statistic is the consistency gap $a_S - i_S$. It is computed separately for guideline feature groups and for suspected-shortcut groups, within strata of the shortcut variables. The interventions stay on the data manifold, so they defeat the off-manifold tricks of Anders et al. by construction.

We hypothesize the following trade-off. An adaptive attacker can shrink the gap only by making the model genuinely respond to guideline features on realistic inputs, which reduces its reliance on the shortcut. We will measure this "evasion cost" directly, by analogy to FRAUD-Detect's forced parity gap.

Research Program

The first component is a red-team benchmark on MIMIC-IV in-hospital mortality, using the cohort and 164-variable setup of MIMIC-IF. Ground truth comes from planting shortcuts. We inject a synthetic site or administrative token correlated with mortality, so we know exactly how much the model relies on it. We also study naturally occurring candidates such as insurance type, which MIMIC-IF already audits.

We build washed models with three attack families. The first is a surrogate-level attack: a LaundryML-style rule list constrained to cite guideline variables. The second is a model-level attack: an Anders-style penalty that steers gradient and SHAP attributions toward guideline features while the shortcut still drives predictions. The third is a substitution attack in the style of Lakkaraju and Bastani, which reports correlated physiological proxies in place of the shortcut. Honest models are trained on the same data with and without the shortcut. Each attack is also run in an adaptive version that adds our consistency gap, with a differentiable approximation of the conditional resampler, to the attacker's objective.

The second component is the audit itself. It needs a conditional generative model of the auditor's held-out data for resampling, query access to $f$, and the explanation as submitted. It does not need weights, gradients or retraining, which matches the non-cooperative setting where accountability work says current audits fail.

The third component resolves a tension in the file. Secure Diagnostics finds that adversarially robust backbones produce explanations better aligned with anatomy, reaching about 31% overlap with surgeon annotations versus 3–10% for standard models. Anders et al. show that gradient explanations can be hijacked. We include adversarially trained mortality models as targets to test whether robustness makes washing harder, or only makes washed explanations more convincing.

A secondary transfer study applies the same audit to clinical text. We use PubMedBERT emergency-department diagnosis prediction from Revisiting Clinical Outcome Prediction for MIMIC-IV, with a planted note-template token as the shortcut and conditional token resampling as the intervention. This study is explicitly exploratory.

Baselines

The naive reference point is current practice: accept a model if its top-ranked SHAP features match guideline features, as in the Alzheimer's alignment study. We run it on exactly the same honest and washed models, with the same guideline feature list. Under washing it should pass every attacked model, and any audit that does no better than this has no value.

The strongest existing faithfulness test is ROAR with ArchDetect, which MIMIC-IF identifies as the most faithful of 12 methods on MIMIC-IV. It is the strongest competitor because retraining after removing features measures reliance directly. It requires training access, which our auditor lacks, so we give ROAR full access and the same compute budget per audited model. Matching ROAR's detection accuracy without retraining counts as a win; we do not need to exceed it. MIMIC-IF reports ArchDetect as fastest-degrading but gives no per-step degradation figures; no reported result found on Lacuna for ROAR detection of manipulated explanations.

The strongest existing washing detector is FRAUD-Detect. We run it twice. The first run uses demographic groups exactly as published. The second run is a fair adaptation that uses shortcut-variable strata, so that our gain cannot be attributed merely to choosing better groups. Its published evidence is a KL divergence difference above 0.6 between honest and fairwashed surrogates on COMPAS, but no reported result found on Lacuna for FRAUD-Detect on clinical data.

We also compare against tangent-space-projected explanations from Anders et al. as a defense rather than a detector. TSP is the only existing method designed to neutralize off-manifold manipulation. We give it the same GPU budget, noting its reported 48 hours on four P100s for CIFAR-10. We also compare against ShorT, which detects shortcut usage when the model can be retrained with an attribute head.

Two ablations isolate our key ingredient. The first replaces conditional resampling with marginal or zero-filling interventions, which are off-manifold. The second removes stratification by shortcut variables. If off-manifold interventions work as well, the manifold claim is wrong. Beating the baselines means higher detection AUROC (honest versus washed) than FRAUD-Detect and the naive check under adaptive attackers, at a false-alarm rate on honest models no worse than ROAR's.

Why not just…?

Why not just use intrinsically interpretable models? Rule lists are themselves the vehicle for fairwashing: LaundryML builds exactly such lists to hide bias. On MIMIC-IV, deep sequence models report AUROC 0.955 and AUPRC 0.769 (IMV-LSTM, MIMIC-IF), so hospitals will keep deploying them. We will nonetheless train a guideline-constrained rule list and logistic model on the same cohort. If they come within a small margin of the deep models, that result is itself a reportable argument against auditing black boxes at all.

Why not just run ROAR or ShorT? Both measure real reliance and are strong. Both require retraining the model, however, and a vendor submitting a model for approval controls the training. Our experiments give both methods full access. If our query-only audit lags them substantially, the honest conclusion is that regulators should demand training access, and we will report it that way.

Why not just run FRAUD-Detect? It is the closest threat. Its premise that unfair models must show group-wise fidelity divergence may already catch many washed models. Its demographic grouping, however, conflates washing with legitimate age or sex effects, which ShorT shows are real in medicine. It also never intervenes on inputs, so it cannot test a substitution attack that keeps surrogate fidelity uniform across groups. The shortcut-stratified FRAUD-Detect baseline settles whether intervention adds anything beyond better grouping.

Why not just adversarially train the model? Robust training improved explanation alignment on fracture X-rays, but it optimizes output stability under small input perturbations, not the honesty of the explanation pipeline. Robust targets are part of our benchmark, so the experiments will show directly whether washing still succeeds on them.

Decisive Checks

The project succeeds if three things hold. First, the naive guideline-alignment check passes most washed models. Second, our audit separates honest from washed models under the non-adaptive attacks across all three families. Third, under the adaptive attacker, any reduction in detection is accompanied by a measured drop in true shortcut reliance (the evasion cost), assessed against the planted-shortcut ground truth. The idea fails if an adaptive attacker can hold the consistency gap near zero while keeping full shortcut reliance. That would show conditional resampling is itself exploitable, for instance through errors in the auditor's generative model, and we will report it as a negative result. The off-manifold ablation tells us whether staying on the data manifold is what makes the audit work. The robust-model arm tells us whether the "anatomical regularizer" finding extends to resisting deliberate manipulation.