Checking Chest X-rays Against the Patient's Record to Catch Forgeries From Unseen Generators

Generative models can now produce or alter medical scans convincingly enough to fool specialists. On the MedForensics benchmark, a radiologist identified forged MRIs and CTs with only about 72% accuracy (Toward Medical Deepfake Detection: A Comprehensive Dataset and Novel Method). Current detectors look for the traces a particular generator leaves in the pixels. Consumer deepfake research shows that such detectors fail when a new generator or a new compression step removes those traces. We propose a different test. Instead of asking whether the pixels look synthetic, we ask whether this image is plausible for this patient, given everything else the hospital recorded: vital signs, lab results, medications, diagnoses, and demographics.

The detector learns only from authentic image–record pairs and never sees a generator, so in principle it has nothing generator-specific to memorize. It can also catch an attack that pixel forensics cannot catch by construction: substituting a real X-ray from a different patient. The main deliverable is a controlled test showing whether this record-consistency signal holds up on generator families and compression settings the detector never trained on, where artifact detectors are expected to degrade.

Core Claim

For forgeries meant to change a clinical conclusion, such as adding or removing cardiomegaly, effusion, or consolidation, or swapping in another patient's image, a consistency score between a chest X-ray and the same patient's electronic health record (EHR) should generalize across unseen generator families and compression levels. Artifact-based detectors should lose substantial accuracy on the same held-out conditions.

We make a second, falsifiable prediction. Detection power should vary by condition in the same way that multimodal fusion gains vary. CareBench found that X-rays add predictive value for "modality-distributed" conditions such as congestive heart failure and chronic obstructive pulmonary disease, but not for sepsis (When Does Multimodal Learning Help in Healthcare? A Benchmark on EHR and Chest X-Ray Fusion). Where the image and the record share little information, no consistency check can work. So forgeries that fake sepsis-related findings should be the hard case.

Relation to Prior Work

The closest work plays three different roles.

DSKI (Toward Medical Deepfake Detection) is the competitor. It is a CLIP-based detector combining noise-domain and multiscale spatial adapters with a retrieval bank of known real and fake samples. It reaches 91.6% mean accuracy, but only within the MedForensics generator pool. It never uses non-imaging data, so a real image from the wrong patient is invisible to it.

Voice-face matching detection (Voice-Face Homogeneity Tells Deepfake) supplies the mechanism. It pretrains a contrastive model on authentic voice–face pairs from VoxCeleb2 and flags mismatches, reaching 85.13% AUC on DFDC out of distribution. We move that recipe from voice–face pairs to image–record pairs. We also inherit its known weakness, which the authors call "matching collapse": a forger who conditions generation on the record itself can defeat the check. We test that attack explicitly rather than assume it away.

Clinical fusion work shows image–record alignment is learnable but always assumes authentic inputs. Examples include supervised contrastive CT–EHR alignment in PECon and the CareBench harness. CareBench also constrains our design: its "information asymmetry trap" shows that fusion models tend to ignore the image when the record is rich. A consistency score that ignored the image would be useless, so we must show directly that ours does not.

Code-representation research supplies two tools rather than an application domain.

The first comes from CODE-MVP (CODE-MVP), which treats meaning-preserving program rewrites, such as variable renaming or loop swapping, as extra positive views in contrastive training. We reuse the idea directly. Transformations that preserve the clinical content of an X-ray are treated as positives, so the score is trained to ignore them. These include PACS-style JPEG re-encoding, windowing, resampling, and noise at different levels.

The second comes from the controlled comparison in Empirical Study of Transformers for Source Code. That study replaced identifiers with placeholders to see what a model uses once the surface cue is gone. We apply the analogous check to the record. We rerun detection with the record reduced to age and sex only, and with demographics removed. This reveals whether the detector is doing clinical reasoning or merely checking that the image looks like a 70-year-old man, which a forger can trivially match.

The same study, along with Revisiting Point Cloud Shape Classification with a Simple and Effective Baseline, shows that gains often vanish once all methods share a training recipe. This sets the protocol discipline below.

Research Program

Benchmark. We build a tampered-record benchmark on the CareBench matched cohort of 7,149 ICU stays with concurrent EHR and X-ray. Patient-level splits and preprocessing come from the released harness, so cohort choices match a published standard. That matters because the two MIMIC-IV benchmarks in the file, CareBench and Benchmarking Foundation Models with Multimodal Public Electronic Health Records, reached opposite conclusions partly because of different cohort filtering.

Forgeries fall into three families:

  • Localized edits. Diffusion inpainting inserts or removes a finding.
  • Full synthesis. Whole images are generated by text- or label-conditioned models from several generator families: diffusion, GAN, and VAE, mirroring MedForensics' spread.
  • Cross-patient swaps. A real image from a different patient replaces the correct one. We match on age and sex so the swap is not trivially detectable.

Each forgery is then re-encoded at several JPEG quality levels.

Protocol. The protocol is leave-one-generator-family-out and leave-one-compression-level-out. Every learned detector gets identical training data, a fixed hyperparameter search budget, and the same encoder capacity class. A second, separate evaluation reruns every artifact detector on the MedForensics X-ray subset under its own splits, as a sanity check that our reimplementations match reported behavior.

Detector. We train an image encoder $f$ and a record encoder $g$ on authentic pairs only. The record encoder is the EHR Transformer from CareBench. Training uses a contrastive objective in which each patient's record must pick out its own image among other patients' images:

$$\mathcal{L} = -\log \frac{\exp(\langle f(T(x_i)), g(e_i)\rangle/\tau)}{\sum_j \exp(\langle f(T(x_j)), g(e_i)\rangle/\tau)}$$

Here $x_i$ and $e_i$ are patient $i$'s X-ray and record, $T$ is a randomly drawn content-preserving transformation (the CODE-MVP-style positive), $\langle\cdot,\cdot\rangle$ is cosine similarity, and $\tau$ is a temperature. The forgery score is the negative similarity, calibrated against the similarity distribution of authentic pairs. Because every candidate image is authentic, and the only way to tell one patient's X-ray from another's is by content the record can explain, the image encoder cannot solve the task without looking at the image. We verify this with an image-shuffling test.

Baselines

Naive reference. A detector that outputs a constant score, giving AUROC 0.5. We also report a nearest-neighbor check that flags an image when its embedding is far from the patient's own earlier X-rays. This is the "just compare with prior exams" reference, and it only applies to patients with a prior study.

Strongest artifact detector. DSKI, as described in its paper, trained on our training generators with the same data and tuning budget. The cited paper reports 91.6% mean accuracy and 92.2% average precision on MedForensics in distribution. There is no reported result found on Lacuna for DSKI, or any detector, on held-out generator families of EHR-paired chest X-rays.

General-purpose detectors. UniFD and RECCE, run under the same protocol through the standardized DeepfakeBench implementations (DeepfakeBench). These show whether medical-specific artifact modeling matters at all under distribution shift.

Diagnosis-disagreement detector. A chest X-ray finding classifier is trained on authentic images. A forgery is flagged when its predicted findings contradict diagnoses and labs in the record. This is the simplest form of consistency checking and the ablation that isolates our key ingredient: learned whole-record contrastive alignment versus hand-specified label agreement.

Zero-shot vision-language model. GPT-4o mini receives the image and a serialized record and is asked whether they are consistent.

Fused detector. A calibrated average of the DSKI and consistency scores, to test whether the two signals complement each other.

Ablations. We remove the transformation positives. We restrict the record to demographics only. We exclude the time-series portion of the record.

What counts as winning. The consistency detector should exceed DSKI's AUROC on held-out generator families and held-out compression levels by a margin larger than the seed-to-seed spread. It should also detect cross-patient swaps, where artifact detectors should sit near 0.5. If DSKI wins on in-distribution generators, that is expected and not a failure.

Why not just…?

Why not just train the artifact detector on more generators? This is a real threat for any generator family someone thought to include, and in distribution DSKI will probably win. The leave-one-family-out protocol settles whether artifact detectors keep their accuracy on the next, unforeseen family. Consumer deepfake evidence suggests they will not: Pentyala et al. (Generalizing Deepfake Detection to Unseen Forgery Methods) and the voice-face work both document large drops outside the training pool. More generators also cannot help with cross-patient swaps, whose pixels are real.

Why not just compare predicted findings with the recorded diagnoses? This is the most serious competitor, because it is our idea in its simplest form. It is included as a baseline for exactly that reason. If it matches the contrastive detector, we will report the simpler method as the result. The core claim, that record consistency generalizes where artifacts do not, would stand either way. We expect it to fall short on edits that change severity rather than label, and on patients whose record carries physiological evidence, such as fluid balance or oxygen requirement, that was never coded as a diagnosis. The per-condition analysis will show where it does and does not fall short.

Why not just ask a vision-language model whether the image fits the record? The MIMIC-IV foundation-model benchmark found that such models, given textualized records, showed limited effectiveness on length-of-stay prediction. In that task a modular encoder reached 0.8532 AUROC, which points to weakness in reasoning over temporal records. We run the model under the same protocol rather than rely on that indirect evidence.

Why not just compare with the patient's previous X-ray? Many ICU patients have no recent prior study. A forged current study may also legitimately differ from the prior one, which is precisely the clinical change being faked. The prior-exam baseline is reported on the subset where it applies.

Proposed Experiments

Main comparison. The central experiment is the held-out generator-family and held-out compression-level comparison described above. We report AUROC and average precision per family, per compression level, and for swaps, with confidence intervals over five seeds and over patient-level bootstrap resamples.

Per-condition prediction. We stratify detection AUROC by the finding targeted by each edit. Consistency detection should succeed on heart failure and chronic obstructive pulmonary disease and approach chance on sepsis-related edits. It should correlate with CareBench's per-condition fusion gains; its best-case phenotyping gain raised AUPRC from 0.472 to 0.495. A flat profile across conditions would suggest the detector is exploiting something other than clinical content.

Demographic shortcut check. With the record reduced to age and sex, detection of age-and-sex-matched swaps should collapse to near chance. If the full detector does not beat this demographics-only version, the method is only doing a demographic check.

Adaptive attack. A generator is conditioned on the patient's own record to produce a consistent fake, the medical analogue of matching collapse. We expect this to defeat the consistency detector, and we measure how much the fused detector retains.

Fairness check. Following CareBench's evidence that fusion can widen sensitivity gaps, we report true-positive-rate differences across racial and sex subgroups. A detector that flags forgeries more reliably for some groups would itself be an equity problem.

Limitation. The file contains no externally linked image–EHR dataset. Cross-institution transfer can therefore be tested only for the image-only detectors, via MedForensics. Testing it for the consistency detector remains open.