Some of the hardest calls in radiology involve morphological mimicry: two conditions with different biology that look nearly the same on a scan. The standard example is a brain lesion after radiotherapy, which may be radiation necrosis (dead tissue, managed conservatively) or recurrent tumor (needs treatment). Both show up as enhancing masses with surrounding swelling. A deep network trained to separate them faces three problems at once. There are only a few hundred confirmed cases. The minority class is rare. And no patient can ever be scanned "with necrosis" and again "with recurrence," so the model never sees a pair that differs only in diagnosis. With so little contrast to learn from, the model tends to latch onto whatever else separates the classes, such as scanner, site, or lesion size.
We propose to manufacture those missing pairs. An unpaired 3D generator (a model that learns to map images between two groups without aligned examples) is restricted to a mask around the lesion and pushes each real lesion toward the opposite diagnosis. Every voxel outside the mask is left unchanged. The result is a matched counterfactual pair: the same patient, the same anatomy, and the same scanner, with only the lesion's diagnostic appearance changed. A 3D classifier is then trained to explain the difference within each pair, in addition to the usual label loss. The claim is that paired, anatomy-locked counterfactuals give a different and more useful training signal than unpaired synthetic augmentation. They should improve minority-class sensitivity and cross-site robustness enough to move a 3D model from beating general readers toward matching subspecialists. As a side benefit, each prediction comes with a voxel map of what would have to change to flip the diagnosis.
Where this sits
The four directions play different roles here. Mimicry is the application, and it is where expert-level claims are least settled. The research file names radiation necrosis versus recurrence as a case where extreme class imbalance yields high accuracy but poor sensitivity on the class that matters, and notes that most systems are evaluated "in a vacuum" without reader tiers. Several studies motivate the bar we aim for. Zhang et al. argue for comparing models against general practitioners, residents, and attending specialists separately. Yang et al. show a dermatology model beating general practitioners and junior specialists, but not uniformly beating senior experts. Wang et al. use an ensemble 3D CNN to stratify lung adenocarcinoma invasiveness, which is itself a mimicry problem (pre-invasive versus invasive ground-glass nodules), and report outperforming thoracic radiologists. The Alzheimer's study combining voxel-based morphometry with deep learning (Early diagnosis of Alzheimer's disease based on brain morphological changes) states the central risk directly: deep models learn shortcuts instead of anatomy. Our pairs are designed to remove exactly those shortcuts.
Volumetric context is a requirement, not a design choice. Necrotic and recurrent lesions differ in how enhancement and edema are arranged across slices. The research file's discussion of the ImageCLEF tuberculosis overview (Overview of ImageCLEFtuberculosis 2019) and the 3D liver work (3D Deeply Supervised Network for Automatic Liver Segmentation) both argue that slice-wise processing breaks through-plane continuity exactly where it is needed to resolve ambiguous boundaries. The file also lists "translating volumetric data without corrupting surrounding non-targeted anatomy" as largely unsolved. Our generator has to solve that problem in 3D, not slice by slice.
Unpaired translation supplies the method. We build on unpaired volumetric harmonization with conditional latent diffusion. That work compresses 3D MRI with an autoencoder and runs class-conditional diffusion in the latent space to remove scanner differences. A diffusion model is a generator that learns to turn noise into samples by removing it step by step. We keep the architecture but change the target: we translate diagnosis rather than scanner, and only inside a lesion mask. Structure-preserving translators such as ContourDiff and curvilinear structure-preserving translation hold contours fixed across the whole image. We deliberately let the lesion change and freeze everything else. Cycle-consistent models (DualGAN) constrain us in a different way. Semi-supervised Cycle-GAN documents that such generators hide high-frequency signals in their outputs, which a downstream classifier could exploit. This failure mode drives a key design choice below. Note that the comparison section of the research file was truncated. I found no work in it that generates diagnosis-flipping, anatomy-locked 3D pairs for training a mimic classifier, but I cannot rule out work outside the file.
Core Claim
Training on matched counterfactual pairs teaches the classifier the direction in feature space that separates the two diagnoses, with patient, site, and anatomy held fixed. Unpaired synthetic samples only add more points to each class. If that holds, paired training should beat equally sized unpaired synthetic augmentation, and the gap should be largest on held-out sites and on the minority class, where shortcuts do the most damage.
Research Program
The generator is a 3D latent diffusion model conditioned on diagnosis. It is trained on all real lesions from both classes plus unlabeled post-treatment scans. To make a counterfactual, we encode a real scan $x$ and partially noise its latent. We then denoise with the opposite label only inside a dilated lesion mask $M$ produced by a pretrained 3D segmenter such as SegVol, and reimpose the original latent outside $M$ at every step. This yields $x'$.
To stop the classifier from detecting "was this generated," we never pair a real scan with a synthetic one. Instead, $x$ is also passed through the same noise-and-denoise procedure with its own label, producing a reconstruction $\tilde{x}$. Both members of the pair, $(\tilde{x}, x')$, then carry identical generator artifacts. The noising strength sets a trade-off between how far the lesion can move and how faithful it stays. We choose it using an independent real-data classifier that should flip, combined with an outside-mask change that should stay near zero.
The classifier $f$ is a 3D residual network in the style of Korolev et al.. Its loss is
$$\mathcal{L} = \mathcal{L}_{\text{CE}}(x, y) + \lambda \, \mathcal{L}_{\text{CE}}\big(g(f(x') - f(\tilde{x})),\, \mathbb{1}[y' > y]\big),$$
where $y$ is the true label, $y'$ the counterfactual label, $f(\cdot)$ the penultimate features, $g$ a linear head that reads the direction of a feature difference, and $\lambda$ a weight. The second term forces the network to encode diagnosis as a consistent direction that survives when everything else is held equal.
At test time, we turn the counterfactual difference $x' - \tilde{x}$ into a 3D "what would flip it" map. Blinded neuroradiologists then judge whether these maps point to recognized imaging signs.
We validate the counterfactuals before using them. The checks are: change outside the mask (expected at the level of autoencoder reconstruction error); flip rate under an oracle classifier trained only on real data; and a blinded two-alternative realism test in which subspecialists try to tell counterfactuals from real lesions. If experts can easily spot counterfactuals, or the oracle does not flip, the training claim is untestable and we report that outcome.
Baselines
The naive reference is a 3D residual classifier trained on real data with class-balanced oversampling and standard spatial and intensity augmentation. Every other system has to beat it. It uses the same backbone, data splits, and hyperparameter search budget as our model.
The strongest competitor on this question is the same backbone trained with an equal number of unpaired synthetic lesions. These come from the identical diffusion model, sampled per class, without the pair term. This is the obvious way to use a generator for class imbalance, and it shares every ingredient except pairing, so it isolates the claim. To beat it, our model must improve minority-class sensitivity at 90% specificity on held-out sites, with confidence intervals from patient-level bootstrap that exclude zero difference.
As the strongest published architecture, we run the ensemble 3D CNN approach of Wang et al., retrained on our data with a matched compute budget. I found no reported result on Lacuna for radiation necrosis versus recurrence. The file also gives no numeric metric for Wang et al.'s invasiveness task, so the baseline numbers will be the ones we measure.
Three ablations isolate the key ingredients. The first translates the whole volume instead of the masked lesion, which tests anatomy-locking. The second pairs a real $x$ with its counterfactual instead of with $\tilde{x}$, which tests the artifact-matching guard. A large gain here would suggest a shortcut, not a real improvement. The third uses a slice-wise 2D generator and classifier, which tests whether volumetric context matters.
Why not just…?
Why not just oversample and augment harder? This is the cheapest fix and may already capture most of the achievable gain if the classes really are separable from the data at hand. The naive baseline tests it directly. Oversampling repeats the same confounded examples, so we expect it to help in-site AUC but not held-out-site sensitivity. If it matches us on held-out sites, the paired signal adds nothing and we will say so.
Why not just use synthetic lesions as extra data, which generative augmentation already does? This is the real threat, and it is why the unpaired-synthetic baseline uses our own generator. Pairing is only worth its complexity if the two differ on held-out sites. A tie means the benefit came from more samples, not from matched contrast.
Why not just use a 2D ImageNet-pretrained model on the most representative slice? Pretraining helps in small cohorts, and the file warns that 3D networks overfit cohorts of hundreds. But the distinguishing signs of mimicry are distributed across slices, the point made in the ImageCLEF tuberculosis overview. The 2D ablation, with both pretrained and scratch 2D models included, settles whether that matters here.
Proposed Experiments
The primary setting is a retrospective, multi-institution cohort of post-radiotherapy brain lesions with diagnosis confirmed by pathology or longitudinal follow-up. Whole sites are held out for testing. No public benchmark for this task appears in the research file. As a replication, we repeat the pipeline on the lung ground-glass invasiveness task studied by Wang et al. to see whether the effect transfers to a second mimic pair and to CT.
The metrics are AUC, minority-class sensitivity at fixed specificity, and calibration. Each is reported per held-out site.
The decisive checks are three. First, paired training must beat unpaired synthetic augmentation on held-out-site sensitivity. Second, the real-versus-$\tilde{x}$ ablation must not produce a suspiciously larger gain, since that would indicate generator fingerprinting. Third, a reader study following Zhang et al. compares the model against residents, neuroradiology fellows, and attending neuroradiologists on the same held-out cases. The proposal succeeds if the model significantly beats residents and is non-inferior to attendings on minority-class sensitivity, and if its flip maps are rated as pointing to plausible imaging signs more often than saliency maps from the naive baseline.
References
- Design and Assessment of Convolutional Neural Network Based Methods for Vitiligo Diagnosis
- Clinical Skin Lesion Diagnosis Using Representations Inspired by Dermatologist Criteria
- An ensemble deep learning model for risk stratification of invasive lung adenocarcinoma using thin-slice CT
- Early diagnosis of Alzheimer's disease based on brain morphological changes: A comprehensive approach combining voxel-based morphometry and deep learning
- Overview of ImageCLEFtuberculosis 2019 - Automatic CT-based Report Generation and Tuberculosis Severity Assessment
- 3D Deeply Supervised Network for Automatic Liver Segmentation from CT Volumes
- Unpaired volumetric harmonization of brain MRI with conditional latent diffusion
- ContourDiff: Unpaired Medical Image Translation with Structural Consistency
- Curvilinear structure-preserving unpaired cross-domain medical image translation
- DualGAN: Unsupervised Dual Learning for Image-to-Image Translation
- Semi-supervised Cycle-GAN for face photo-sketch translation in the wild
- SegVol: Universal and Interactive Volumetric Medical Image Segmentation
- Residual and plain convolutional neural networks for 3D brain MRI classification