Networks that enhance medical volumes, for example by predicting a contrast-enhanced scan from a non-contrast one, are trained with losses that average over every voxel (every 3D pixel). Some of these networks also add a term that makes neighbouring slices agree with each other. Both choices look harmless, but in practice they hand almost all of the training signal to healthy tissue, and small lesions end up receiving very little. A lesion that occupies well under 1% of a volume and appears on only two or three slices contributes almost nothing to a voxel-averaged loss. It also looks, to a slice-agreement term, exactly like the kind of abrupt slice-to-slice change the term exists to remove. We suspect that current volumetric enhancement pipelines are therefore quietly trained to fade the findings a radiologist most needs to see, while their global image-quality scores stay high.
We propose to replace the voxel-averaged reconstruction and slice-agreement losses with balanced versions, adapted from a segmentation loss designed for extreme class imbalance. Lesion voxels are weighted equally with an equal-sized set of the hardest non-lesion voxels, and the easy background bulk is discounted. The through-plane term is changed from "neighbouring slices should look alike" to "the change between neighbouring slices should match the change in the real target." Lesion masks are needed only during training, never at inference. The claim is testable and could fail. At matched global quality, balanced training should preserve the contrast of small, few-slice lesions better than standard training. It should also create fewer lesion-like bright spots in healthy tissue than a simple mask-weighted loss.
Where this sits in prior work
The method we build on is 2S-CEDiff, which synthesizes contrast-enhanced CT from non-contrast CT in two stages. The first stage is a 2.5D diffusion model working on a 5-slice window. The second is a lightweight 3D U-Net refiner trained with pixel L1 loss, a Z-axis SSIM consistency loss and a volumetric SSIM loss. It reports 29.87 dB PSNR and 0.891 SSIM on HeartCT. Its evaluation covers heart chambers, aorta and myocardium, all of which are large structures. Our proposal keeps its architecture and changes only the second-stage objective, so any difference we measure can be attributed to the loss.
The balancing technique comes from Balance Loss. That work shows that summing many small soft background probabilities biases segmentation training. Its fix is to mine the hardest background voxels, equal in number to the foreground voxels, and to gradually decay the weight on easy voxels. It raises KiTS19 Dice from 59.8 to 66.8. Balance Loss is a classification objective. Carrying it into a regression setting, and into a through-plane term, is the new step here. It also inherits a known cost: Balance Loss needs staged scheduling to train stably, and we plan for that.
Three papers motivate the concern that lesion-level failures can hide behind good global numbers:
- Pancreas segmentation with 3D Gaussian smoothing raised Dice from 56% to 68% by enforcing cross-slice continuity. This shows that continuity helps large organs. It says nothing about structures that are genuinely brief along the depth axis.
- The 2D DeepLesion pipeline reports that full-image nnUNet reaches only 34.1% Dice on tiny lesions. Its own single-point-of-failure analysis shows that one missed slice loses the lesion entirely.
- SegGuidedDiff warns that standard image-quality scores miss anatomically wrong outputs.
Generative rebalancing methods such as Red-GAN and CPGGAN address imbalance by synthesizing extra minority samples in 2D. CPGGAN reports that false positives spike at large synthetic volumes. That result is a caution for us too: pushing a model toward lesions can create phantom lesions. For this reason we treat false enhancement in healthy tissue as a primary metric, not an afterthought.
Core Claim
Let $\hat{y}$ be the refined volume and $y$ the real target. Let $F$ be the set of lesion voxels, taken from training masks. Let $H$ be the $|F|$ non-lesion voxels with the largest current error. The balanced reconstruction loss is
$$ \mathcal{L}_{\text{rec}} = \frac{1}{|F|}\sum_{v\in F} |\hat{y}_v - y_v| + \frac{1}{|H|}\sum_{v\in H} |\hat{y}_v - y_v| + \alpha_t \frac{1}{|B|}\sum_{v\in B} |\hat{y}_v - y_v|. $$
Here $B$ is all voxels, and $\alpha_t$ decays over training as in Balance Loss, so the easy bulk keeps some weight early on and fades later. The balanced through-plane loss applies the same three-way weighting to $|\Delta_z \hat{y}_v - \Delta_z y_v|$, where $\Delta_z$ is the intensity difference between a voxel and the voxel directly below it in the next slice. This term rewards reproducing a real lesion's abrupt entry and exit along the depth axis rather than smoothing it away.
The hard set $H$ does two jobs. First, it is where the model tends to invent enhancement, such as vessels and bright mimics. Second, it is the part a plain mask-weighted L1 ignores. We therefore predict that $H$ is responsible for any false-positive advantage over mask weighting.
Research Program
The first step is diagnostic. It tests the premise before we build anything. We will take the 2S-CEDiff first stage as published and train its second-stage refiner with the original objective. We then compare lesion contrast-to-background ratio in the refiner's output against the first-stage output, stratified by how many slices a lesion spans. If refinement lowers contrast specifically for lesions spanning three or fewer slices, the erasure hypothesis holds. If it does not, the contribution shrinks to the gradient-budget argument, and we will report the work that way.
The second step trains refiners under several objectives, all on the same frozen first stage, with the same compute and tuning budget:
- the original objective;
- mask-weighted L1 with the original Z-SSIM;
- our balanced reconstruction loss with the original Z-SSIM;
- our full balanced reconstruction plus balanced through-plane loss.
The third step checks whether lesion-level differences matter downstream. A segmenter trained once on real target images and then frozen is run on each model's synthetic volumes, and we measure lesion recall.
Data follows the file's benchmarks. The primary setting is BraTS2021. Here we synthesize the contrast-enhanced T1 sequence from the non-contrast T1 sequence, the MRI analogue of non-contrast to contrast CT, and the enhancing-tumour label serves as the small minority region. 2S-CEDiff already reports being evaluated on BraTS2021, but we found no reported result on Lacuna for 2S-CEDiff's numbers on BraTS2021, so we will reproduce its baseline ourselves. HeartCT, which has no lesions, acts as a regression guard: the balanced losses must not cost global quality relative to the reported 29.87 dB and 0.891. Coltea-Lung-CT-100W provides a second CT check of global fidelity across contrast phases.
Baselines
Naive reference. The simplest reference is the first-stage 2.5D output with no 3D refinement. It tells us whether refinement helps or hurts lesions at all.
Strongest existing method. The main competitor is 2S-CEDiff, which combines 2.5D diffusion with 3D refinement using L1, Z-axis SSIM and volumetric SSIM. It is the strongest competitor because it is the only method in the file that explicitly targets volumetric consistency during enhancement. Its reported numbers are PSNR 29.87 and SSIM 0.891 on HeartCT; no reported result was found on Lacuna for its BraTS2021 scores. We run it with its published loss weights, and we also give it the same hyperparameter search budget as our variants. Beating it means one of two outcomes on BraTS2021, with non-inferior global PSNR and SSIM (within 0.3 dB and 0.005 respectively). The first is a significantly higher enhancing-tumour contrast ratio and higher downstream lesion recall for lesions spanning three or fewer slices. The second is fewer false-positive enhanced components in healthy tissue.
Mask-weighted L1. This baseline weights lesion voxels to equal the background in total. It uses the same masks as our method and the same budget. It is the ablation that isolates the hard-voxel set $H$.
Balanced reconstruction, original through-plane term. This ablation isolates the balanced through-plane loss.
Segmentation directly on the source images. A reference that skips enhancement entirely and segments the non-contrast input with Balance Loss.
Why not just…?
Why not just weight the L1 loss by the lesion mask? This is the most serious alternative, and it may capture most of the lesion-fidelity gain. Our prediction is narrower. Mask weighting pushes intensity toward lesions, but it gives no extra signal to lesion-like healthy tissue. As a result it should increase false enhancement in vessels and bright mimics, the same failure mode CPGGAN saw when it overdid augmentation. The mask-weighted baseline, run with identical masks and budget, settles this. If it matches our method on both lesion recall and false positives, we will conclude that hard-voxel mining is unnecessary and report exactly that.
Why not skip enhancement and segment the non-contrast scan directly with Balance Loss? For pure detection this may well be competitive, and we include it as a reference. However, enhancement outputs are read by radiologists, not only by segmenters. The question we are asking is whether the images themselves remain faithful. The direct-segmentation reference shows how much diagnostic information the synthetic image preserves relative to the source image.
Why not enhance slice by slice with a strong lightweight 2D enhancer such as MGN and then smooth across slices? MGN reaches 25.59 dB on MIT-Adobe-5K with 0.41M parameters, but it has no through-plane information. Post-hoc smoothing is exactly the operation we hypothesize erases few-slice lesions. We will include a 2D-MGN-plus-Gaussian-smoothing variant in the step-one diagnostic. It should show the same size-dependent contrast loss, only more strongly.
Proposed Experiments and Decisive Checks
All BraTS2021 results will be reported stratified by lesion volume and by the number of slices each lesion spans. Metrics are lesion-region PSNR and SSIM, lesion contrast-to-background ratio, frozen-segmenter lesion recall, and false-positive enhanced components per volume, alongside global PSNR, SSIM and Z-axis SSIM. We use three seeds and paired bootstrap tests over patients.
Two findings would decide the question. The first is whether standard Z-SSIM refinement measurably fades few-slice lesions; this is the premise. The second is whether the full balanced objective beats mask-weighted L1 on false positives at equal lesion recall; this is the key ingredient. If the first fails, the paper becomes a negative result about through-plane losses. If the second fails, the simpler mask-weighted loss is the recommendation.
References
- 2S-CEDiff: A Two-Stage Diffusion Framework for Generating High-Fidelity Contrast-Enhanced CT Images from Non-Contrast Scans
- A Unified Loss for Handling Inter-Class and Intra-Class Imbalance in Medical Image Segmentation
- Deep convolutional networks for pancreas segmentation in CT imaging
- A Unified 2D Framework for DeepLesion Detection, Segmentation and Short Report Generation
- Anatomically-Controllable Medical Image Generation with Segmentation-Guided Diffusion Models
- Red-GAN: Attacking class imbalance via conditioned generation. Yet another medical imaging perspective
- Learning More with Less: Conditional PGGAN-based Data Augmentation for Brain Metastases Detection Using Highly-Rough Annotation on MR images
- Mutual Guidance and Residual Integration for Image Enhancement