Image-to-image translation networks for medicine, such as those that synthesize a T2-weighted MRI from a T1 scan, usually train against a patch discriminator. This is a critic network that scores many small image windows as real or fake, rather than scoring the whole image once. The size of those windows, the critic's receptive field, is almost always inherited from defaults such as 70×70 or 32×32 pixels. Those defaults were tuned on datasets far larger than a typical clinical cohort. We argue that when training data are scarce, the receptive field is the most important knob on the discriminator, because it controls two things at once. It sets what the critic can see: long-range anatomical errors escape a small window. It also sets how many roughly independent examples the critic learns from: a large window yields few examples, which it can memorize. The best window should therefore shrink as the cohort shrinks. It should also be predictable from the data before any adversarial training is run.
We propose a cheap patch-size probe and test whether it can pick the critic's receptive field, and weights across several scales, for a given cohort size. The probe trains small real-versus-regression-output classifiers at several window sizes and reads off two curves: held-out accuracy, and the gap between training and held-out accuracy. We then ask whether the critic chosen by the probe matches an exhaustive sweep and beats both the default critic and the strongest few-shot baseline at 20 to 300 training slices. The contribution is a tested account of why patch critics help in small-data regimes, plus a procedure that replaces guesswork with a measurement.
Core Claim
The theory motivating this work is the subadditivity result of GANs with Conditional Independence Graphs: On Subadditivity of Probability Divergences. It proves that when data follow a known conditional-independence graph, the divergence between two joint distributions is bounded by a sum of divergences over local neighborhoods. As a result, local critics need a number of samples that grows with the neighborhood size rather than with total dimension. That paper tested the idea only on 5- to 20-node graphs and a 15-dimensional trajectory, and it assumes the graph is known.
Images have no known graph, but the result suggests a trade-off we can measure. The critic's excess error at window radius $r$ should behave like $B(r) + V(r, N)$. Here $B(r)$ is the part of the real-versus-generated discrepancy that lives at scales larger than $r$, which no window of that size can detect. $V(r, N)$ is the estimation error of a critic whose input dimension grows like $r^2$, trained on $N$ images that each supply roughly $(H/r)^2$ weakly dependent windows, where $H$ is the image side length. The dependence on $r$ and $N$ follows the usual pattern that estimation error grows with input dimension and falls with effective sample count. We do not claim a tight constant.
The testable predictions are threefold. First, the best radius $r^*(N)$ grows with $N$. Second, at small $N$ the default 70×70 critic sits on the memorization side of the trade-off. Third, both $B(r)$ and $V(r,N)$ can be estimated before GAN training from held-out and training accuracy of small probe classifiers.
What Prior Work Does and Leaves Open
The closest system is SRU-Pix2Pix. It trains on about 300 BraTS 2023 slices with a "simplified PatchGAN" and reports that this setup avoided the over-smoothing and hallucination seen with a Brownian-bridge diffusion model. That paper supplies our main baseline and our evidence that a patch critic is useful in this regime. However, its critic is fixed and hand-simplified, and it does not study scaling below 300 slices. We keep its generator design and vary only the critic.
Two other lines of work shape the design space:
- Critic overfitting under scarcity. One-Shot Image-to-Image Translation via Part-Global Learning pairs a random-crop part critic with a global critic and a balanced loss so the critic does not overfit a single target exemplar. It works on natural images only, and its crop size is chosen by hand.
- Semantic blindness. ALDEN addresses the fact that uniform patch critics treat air, soft tissue and bone identically. It fuses DINOv2 and MedSAM features into the critic and reports 33.71 dB PSNR on Mayo2016. It targets denoising, adds heavy pretrained models, and does not study cohorts below 50.
Both are competitors, not the thing we build. Our proposal changes neither the critic's features nor its loss balancing. It changes only its spatial reach and per-scale weighting, and does so by measurement.
Two results constrain the design. Unveiling COVID-19 from Chest X-ray with deep learning found that ResNet-50 overfit where ResNet-18 did not on small cohorts, which warns that any added critic capacity is suspect. MS-GAN reports locally realistic patches that fail to blend with their neighbors, which warns that shrinking the window too far should produce seams. Our trade-off predicts exactly this failure, and we measure it.
Research Program
The probe works as follows. First, fit the translation generator with only an L1 and MS-SSIM regression loss, as in the non-adversarial part of SRU-Pix2Pix's objective. For each window size on a grid from 16 to 128 pixels, train a small fixed-width convolutional classifier to tell real target patches from regression-output patches. Use half the training patients for fitting and the other half for scoring, split by patient so that slices from one subject never appear on both sides.
Held-out accuracy as a function of window size estimates how much detectable discrepancy each scale can see; this approximates $B(r)$ from the complementary side. The train-minus-held-out accuracy gap estimates $V(r,N)$ directly. The probe then outputs one of two choices:
- the single window size that maximizes held-out accuracy minus the gap, or
- a set of weights over a small multi-scale critic, proportional to each scale's held-out accuracy above chance and penalized by its gap.
The whole probe costs a few classifier trainings, which is far cheaper than one GAN run per candidate window.
We then train full adversarial models with the SRU-Pix2Pix-style generator (2.5D input, SE-residual blocks, nested U-Net++ decoder) and the same composite loss, changing only the critic. The first question is whether the curve of translation quality against window size, traced by an exhaustive sweep, has an interior optimum that moves toward smaller windows as the cohort shrinks. The second is whether the probe's choice lands within one grid step of the sweep optimum.
Baselines
Naive reference: Pix2Pix with the default 70×70 critic. This is standard Pix2Pix with its usual PatchGAN, trained on each cohort size with the same augmentation and iteration budget. On BraTS 2023 T1→T2 with about 300 images, SRU-Pix2Pix reports Pix2Pix at PSNR 24.58 and SSIM 0.8642. This tells us how much of any gain comes from the generator versus the critic.
Strongest existing method: SRU-Pix2Pix with its simplified PatchGAN. It reports PSNR 28.31 and SSIM 0.9281 on BraTS 2023 T1→T2 at about 300 images, and PSNR 24.07 in zero-shot transfer to BraTS 2019. It is the strongest competitor because it was designed for this exact regime and already outperforms CycleGAN, NICE-GAN and BBDM there. We reimplement it and first confirm we reproduce its number within noise at 300 slices, then run it unchanged at 100, 50 and 20 slices. Every other arm shares its generator, losses, optimizer, augmentation, iteration count and five seeds. Beating it means a statistically significant gain in held-out PSNR and LPIPS at 50 slices or fewer, with no loss at 300. Parity at 300 combined with gains at small cohorts would support the claim.
Anatomy-aware critic in the style of ALDEN. This arm fuses frozen DINOv2 features into the critic's intermediate layers, given the same tuning budget. It tests whether semantic content, rather than spatial reach, is the missing ingredient. ALDEN reports 33.71 dB PSNR on Mayo2016 for denoising; for few-shot translation, no reported result was found on Lacuna.
Part-global critic with balanced adversarial loss. This arm uses a global critic plus a random-crop critic with the loss-balancing hyperparameter of the one-shot part-global method, tuned on the same validation patients. For medical translation, no reported result was found on Lacuna.
Oracle and ablations. The oracle is an exhaustive window sweep selected on validation patients, which gives an upper bound for the probe. Uniform multi-scale weighting isolates the probe's weighting. A probe that ignores the generalization gap and uses held-out accuracy alone isolates the claim that memorization, not only visibility, should set the window.
Why not just…?
Why not just sweep the patch size on a validation set? This is a real threat, and it serves as our oracle. With 20 training slices, the validation set is only a handful of patients, and a sweep requires a full adversarial run per window size and seed. The selection is therefore both noisy and costly. If the sweep reliably finds the optimum and the probe does not match it, the method has no value beyond diagnosis, and we will report that. The probe earns its place only if it tracks the sweep optimum across cohort sizes at a small fraction of the compute.
Why not just use a multi-scale critic and let training sort out the scales? This could plausibly work as well, which is why the proposal selects weights rather than a single window. Our trade-off predicts that uniform weighting fails at small cohorts because the large-scale critic memorizes and dominates the generator's gradients. The uniform-weight ablation settles this. If it matches the probe-weighted critic at 20 and 50 slices, the prediction is wrong for this setting.
Why not just shrink or regularize the critic, for example with fewer channels, a balanced loss, or low-rank adapters as in NA-UNETR? Capacity control and receptive-field control are confounded in the default PatchGAN, because shrinking the network also shrinks what it sees. To separate them, we run a width sweep at fixed receptive field alongside the window sweep at fixed width. If width alone recovers the gains, the story is capacity rather than spatial reach.
Why not just use the anatomy-aware critic? ALDEN's evidence comes from denoising, where paired data were scarce but not as extreme as 20 slices. Its added pretrained features also increase the critic's input richness, which our trade-off predicts can raise memorization. The ALDEN-style arm tests this directly, and combining it with the probe-chosen window is a natural follow-up if both help.
Proposed Experiments
Datasets. The main testbed is BraTS 2023 T1→T2, T1→FLAIR and T2→FLAIR, subsampled by patient to 300, 100, 50 and 20 training slices, with BraTS 2019 as a zero-shot test. We add IXI PD→T2 to check that the effect holds on healthy anatomy without tumors. We add Mayo2016 low-dose CT denoising to test transfer to a modality with different texture statistics.
Metrics. We report PSNR, SSIM and LPIPS, and additionally PSNR restricted to the BraTS tumor masks, because the clinical risk named in the file is hallucinated or erased pathology. To check the mechanism directly, we log the critic's train-versus-held-out real-patch accuracy during GAN training, and a seam score (gradient discontinuity at window-sized offsets) to catch the MS-GAN failure mode when windows are too small.
Decisive checks. Four outcomes decide the claim:
- The sweep optimum moves to smaller windows as the cohort shrinks; a flat curve refutes the core claim.
- The probe's choice falls within one grid step of that optimum in most task and cohort cells.
- At 50 slices or fewer, the probe-chosen critic significantly beats the SRU-Pix2Pix critic on PSNR, LPIPS and tumor-region PSNR, without losing at 300.
- The critic's measured generalization gap during GAN training is ordered as the probe predicted.
If the probe-chosen critic beats the defaults but uniform multi-scale does equally well, the conclusion becomes "use multi-scale critics in small cohorts," and the probe remains only a diagnostic.
References
- GANs with Conditional Independence Graphs: On Subadditivity of Probability Divergences
- SRU-Pix2Pix: A Fusion-Driven Generator Network for Medical Image Translation with Few-Shot Learning
- One-Shot Image-to-Image Translation via Part-Global Learning With a Multi-Adversarial Framework
- Anatomy-Aware Low-Dose CT Denoising via Pretrained Vision Models and Semantic-Guided Contrastive Learning
- Unveiling COVID-19 from CHEST X-Ray with Deep Learning: A Hurdles Race with Small Data
- Multi-sequence generative adversarial network: better generation for enhanced magnetic resonance imaging images
- A Neighborhood Attention Transformer Network for Enhanced 3D Segmentation of the Left Anterior Descending Artery