Artifact-Matched Counterfactual Negatives for Rare Findings in Chest X-rays

Medical image classifiers are trained on data where healthy or common cases vastly outnumber rare findings. They respond by learning the majority class, and sometimes by learning shortcuts such as pacemakers or scanner tags instead of the disease itself. One way out is to manufacture rare-class examples from the abundant common ones. An image editor adds a lung lesion to a normal radiograph of a real patient, a counterfactual edit, and the classifier then sees the same patient with and without the finding. Two chest X-ray papers already generate such counterfactuals, but both simply pour the synthetic images into ordinary training. The NLP literature shows that this invites the model to fixate on whatever the edit changed. In images, the edit also carries generator fingerprints that only edited samples have, so the classifier can learn "synthetic means diseased."

This note proposes a specific fix and a test of whether it matters. For each abundant normal image $x$, we produce two outputs from the same diffusion editor. The first is a "null edit" $\tilde{x}$, which runs the same invert-and-regenerate pass with no change requested. The second is a pathology edit $c$ that adds the rare finding. Automated referee models check that $c$ is still the same patient, keeps the same apparent sex and devices, and actually shows the finding. The classifier is trained so that $x$ and $\tilde{x}$ sit together in feature space and $c$ is pushed away. Because $\tilde{x}$ and $c$ carry identical generator artifacts, the only signal separating them is the pathology.

The core claim is that pairing each pathology counterfactual with a null-edited twin, and training on the resulting triplet with a contrastive term, improves rare-finding detection on real held-out images. The claim is measured against pooling the very same verified counterfactuals into cross-entropy training. It is also measured against the obvious rebalancing tricks. If the pooled version does as well, the contribution is only the referee-gated generator, and we will say so.

What exists and where this sits

Three papers are the closest relatives, and none of them does this.

PRISM fine-tunes the U-Net of Stable Diffusion on CheXpert. It edits radiographs by inverting them and changing the prompt, with DDIM plus null-text inversion, which recovers the noise that regenerates a given image. It retrains an EfficientNet on the edits and reports Pleural Effusion accuracy rising from 0.80 to 0.88. PRISM also documents the failure modes that constrain our design. Removing devices sometimes shifted apparent sex, and removing cardiomegaly also erased pacemakers. We reuse PRISM's editing method, but we add referees that reject exactly these failures, and we change how the edits are consumed.

Causal Disentanglement for Robust Long-tail Medical Image Generation generates rare-pathology counterfactuals with a disentangled latent diffusion model. It reports a MIMIC-CXR FID of 7.24 and improved F1 for Consolidation and Pneumothorax, but gives no F1 value that we could find. It also pools images into cross-entropy training and spends 2–3 minutes per sample optimizing the diffusion noise seed. We treat it as evidence that pathology counterfactuals help rare classes, and as a cost ceiling we aim to stay well under.

Robust image representations with counterfactual contrastive learning (CF-CL) is the method-supplying paper. It shows that using a counterfactual twin as an explicit contrastive pair beats adding unpaired synthetic data. The reported gains are up to 3% ROC-AUC on under-represented scanners and 4–6% on the external VinDR-Mammo set. However, CF-CL intervenes only on scanner or sex and uses the twin as a positive, meaning same label. We intervene on pathology and use the twin as a negative. This is the direction that PairCFR found necessary in text, where training jointly on cross-entropy and a contrastive loss over counterfactual pairs raised out-of-distribution NLI accuracy for BERT from 52.69% to 55.59%.

Neither CF-CL nor PairCFR had to deal with a generator whose artifacts correlate with the label. The null-edit twin is our answer to that problem, and it has no counterpart in either paper.

Two further papers supply the verification design and warn about metrics. When the Edit Changes the Patient showed that realistic-looking edits can silently change identity. In its study, clinical readers judged SDEdit pairs to be the same eye only 36% of the time, against 87% for real pairs. It proposed automated referees, namely a same-eye verifier and a sex classifier, but never closed the loop into training. Benchmarking Counterfactual Image Generation showed that GANs fail the null-intervention sanity check. This is why we run the null edit through the same editor rather than using the raw image as the "no-change" reference.

The text-side analogue, CAISA, used random verb replacement for a minority clinical class. Its edits were unverified, and performance degraded as the number of augmented samples grew. That is a cautionary result we will test directly by scaling the number of pairs.

Research Program

The generator is a PRISM-style Stable Diffusion model with a fine-tuned U-Net, trained on the full CheXpert training split. This deliberately accepts the "pretraining paradox" from GAN-based Data Augmentation for Chest X-ray Classification. The generator learns from the whole corpus, but the classifier sees only a small labeled minority set. Chest radiography is precisely the regime where a large pool of unlabeled or normal images exists alongside rare labeled findings.

For each normal source image, we run inversion once and regenerate twice. One pass uses the unchanged prompt, which yields $\tilde{x}$. The other adds the target finding, for example "with a lung lesion in the right upper zone," which yields $c$. Optionally, a spatial mask confines the edit to one lung region. We will ablate the mask rather than assume it helps.

Three referees gate each pair, and all are trained on data separate from the evaluation split.

The first is a same-patient verifier. It is a Siamese network trained on CheXpert's multiple studies per patient, and it checks both ($x$, $\tilde{x}$) and ($x$, $c$).

The second is a set of attribute preservers: a sex classifier and a support-device detector. They must agree between $\tilde{x}$ and $c$, which directly targets PRISM's reported drift.

The third is a flip verifier. It is an ensemble scoring the target finding, and it must rate $c$ positive and $\tilde{x}$ negative.

The flip verifier has an obvious circularity for rare classes. A detector trained on few positives is a weak judge of positives. We address this in two ways. We train the flip verifier on all available labeled positives, while the downstream classifier is restricted to a subsampled regime. We also have a radiologist audit a random sample of accepted and rejected pairs, so that we can report how often the referees agree with a human. That audit replaces per-sample human annotation, which is the scalability point.

Training uses cross-entropy on real and synthetic images plus a pairwise term. For an anchor embedding $z_x$, the twin $z_{\tilde{x}}$ is the positive, the edited $z_c$ is a forced negative, and the other in-batch images $z_j$ are additional negatives:

$$ \mathcal{L} = \mathcal{L}_{CE} - \lambda \log \frac{\exp(s(z_x, z_{\tilde{x}})/\tau)}{\exp(s(z_x, z_{\tilde{x}})/\tau) + \exp(s(z_x, z_c)/\tau) + \sum_j \exp(s(z_x, z_j)/\tau)} $$

Here $s$ is cosine similarity, $\tau$ is a temperature, and $\lambda$ weights the pairwise term. The triplet ($x$, $\tilde{x}$, $c$) is always placed in the same batch, following PairCFR. PairCFR found that ambiguous classes, specifically NLI's "Neutral", hurt contrastive counterfactual training. By analogy, CheXpert's uncertain labels are excluded from sources and anchors. They are not converted into U-Ones.

Baselines

The naive reference is empirical risk minimization, ordinary training on the imbalanced data with a DenseNet-121 or ResNet-50 classifier. It is run on the same low-data CheXpert subsets used by the GAN augmentation paper (1% and 10% of labels), with identical architecture, epochs, and hyperparameter search budget across all arms. Beating it is necessary but uninformative.

The second baseline is class-balanced loss weighting combined with random minority oversampling, tuned over the same budget. This is the cheapest serious competitor. The claim that oversampling memorizes positives, attributed to Mitigating Class Imbalance in Clinical Machine Learning, is an argument, not a measured number for our setting, so we must measure it.

The strongest published augmentation baseline with numbers in this regime is conditional-GAN augmentation from GAN-based Data Augmentation for Chest X-ray Classification. At 1% data it reports ROC-AUC rising from 0.727 to 0.758 for Lung Lesion, from 0.566 to 0.594 for Pleural Other, and from 0.583 to 0.656 for Fracture. We target these three findings precisely so that this comparison is direct. We will rerun their protocol on our splits rather than rely only on the quoted figures. Beating it means a higher real-test ROC-AUC on each of the three findings at 1% data, averaged over at least five seeds, with non-overlapping bootstrap intervals for at least two of the three.

The most important baseline is an ablation that isolates our ingredient. It is PRISM-style pooling, meaning the identical referee-accepted counterfactuals $c$ added to cross-entropy training with no pairing. The same number of synthetic images, the same generator, and the same compute are used. A second ablation keeps the contrastive term but uses the raw image $x$ instead of the null-edit twin $\tilde{x}$ as the positive. This separates the benefit of contrast from the benefit of artifact matching.

A CF-CL-style arm uses scanner or sex twins as positives only, which tests whether pathology negatives add anything over invariance pairs. A final arm drops the referees and accepts every pair, which measures what verification buys.

Why not just…?

Why not just reweight the loss or oversample? It costs nothing, and for moderately rare findings it may be enough. That is a real threat. The heart-sound CNN in Classification of Heart Sound Recordings Using Convolution Neural Network shows the characteristic skew under imbalance, with 70.8% sensitivity against 88.2% specificity, but it did not test reweighting. Our reweighting arm, evaluated as sensitivity at fixed specificity on real test images, settles the question for each finding. We expect generated pairs to matter most at the 1% regime, where reweighting can only repeat a few dozen positives.

Why not just pool the counterfactuals into cross-entropy, as PRISM did? This is the most likely way our proposal fails. PRISM's gain on effusion shows pooling works to some degree. The evidence that pairing adds value comes from other settings: text in PairCFR, and scanner invariance in CF-CL's comparison against "Contrastive+." The pooled-counterfactual ablation uses identical images and compute. If it matches the triplet objective on real-test ROC-AUC and on external MIMIC-CXR, we drop the pairing claim.

Why not just generate minority images from scratch, with no source patient? Unpaired generation removes the identity and artifact controls that make the contrast clean. Separately, Benchmarking Counterfactual Image Generation found that GANs alter non-targeted features, which bears on generator faithfulness rather than on pairing itself. The GAN baseline above is this alternative, so it is tested rather than argued away.

Proposed Experiments

The main experiment trains all arms on CheXpert subsets at 1%, 10%, and 100% label fractions for Lung Lesion, Pleural Other, Fracture, and Pleural Effusion. We evaluate on the real CheXpert test set and, without retraining, on MIMIC-CXR and PadChest for transfer.

There are four decisive checks.

The first is an artifact probe. We train a linear classifier on frozen features to tell $\tilde{x}$ from real $x$. If the triplet model's features separate them less than the pooled model's features do, the null twin is doing its job.

The second is a device-stratified evaluation. We score rare findings separately on images with and without support devices, which tests whether gains come from shortcut reinforcement.

The third is a scaling curve over 100 to 10,000 accepted pairs. CAISA saw degradation beyond 400 samples, and a flat or declining curve in our pooled arm but not in the triplet arm would be strong evidence for pairing.

The fourth is a cost and audit report: GPU-seconds per accepted pair, referee acceptance rate, and radiologist agreement with referee decisions on a sample of a few hundred pairs. We aim to stay far below the 2–3 minutes per sample of the Causal Disentanglement method.

ISIC 2019 serves as a secondary test in a second modality. We found no reported result on Lacuna for any baseline on ISIC 2019, so there we compare only against our own reruns.