Which Way Should the Arrow Point? Asymmetric Synthetic-to-Real Feature Mapping for Generative Augmentation

Adding images from a generative model to a training set sometimes helps a classifier and sometimes hurts it. On CIFAR-10, images from a generative adversarial network (GAN, a generator trained against a discriminator that tries to tell fake images from real ones) lowered ResNet18 accuracy when added on top of standard augmentation. Images from a diffusion model (a generator that learns by gradually denoising random noise into samples) raised it (Toward Understanding Generative Data Augmentation). The usual responses work on the generator: train a better one, filter its outputs, or align it to an expert. We propose leaving the generator alone and changing how its samples enter training. Synthetic images pass through a small learned "mapper" that projects their features into the feature space of real images. The real features are held fixed as an anchor, so the alignment loss can never move them. The mapper is discarded at test time.

The idea is borrowed from domain adaptation, where asymmetric methods freeze one domain's representation and force the other to conform to it. This avoids the "moving target" failure, in which an unreliable domain drags the reliable domain's well-separated class clusters into a blurred compromise (Mind the Discriminability: Asymmetric Adversarial Domain Adaptation). Our claim is that generative augmentation is a domain-adaptation problem with a known reliability ordering. Real data is faithful but scarce. Synthetic data is plentiful but carries generator artifacts. The arrow should therefore point from synthetic to real, at least once there is enough real data to define a stable anchor.

Core Claim

Let $f$ be the backbone, $h$ the classification head and $g$ a lightweight residual mapper that starts near the identity. A real image $x$ is classified as $h(f(x))$. A synthetic image $\tilde{x}$ with generator label $\tilde{y}$ is classified as $h(g(f(\tilde{x})))$. Training minimizes

$$ \mathcal{L} = \mathrm{CE}(h(f(x)), y) + \gamma\,\mathrm{CE}(h(g(f(\tilde{x}))), \tilde{y}) + \lambda \sum_{c} \big\| \bar{g}_c - \mathrm{sg}[\mu_c] \big\|^2 . $$

The terms are defined as follows. $\mathrm{CE}$ is cross-entropy. $\bar{g}_c$ is the batch mean of mapped synthetic features for class $c$. $\mu_c$ is an exponential moving average of real features for class $c$. $\mathrm{sg}$ is a stop-gradient, so the alignment term moves only $g$ and the synthetic path through $f$, never the real prototypes. $\gamma$ and $\lambda$ are scalar weights.

The mapper is meant to absorb artifacts that are systematic for a given generator. This spares the shared head from bending its decision boundaries to fit them. The backbone still learns from synthetic diversity through the synthetic cross-entropy term.

We make two predictions. First, with an imperfect generator in a regime with plenty of real data, this design removes the accuracy loss that naive mixing causes. Second, there is a crossover in the number of real samples $m$. When $m$ is very small, the real prototypes $\mu_c$ are too noisy to serve as an anchor. In that regime the reverse direction should win: real features mapped into a fixed synthetic space, as in Feature Mapping for Learning Fast and Accurate 3D Pose Inference From Synthetic Images. The crossover itself, and not just a gain, is the scientific result. It would give practitioners a rule for which side to anchor.

Relation to Prior Work

Several papers motivate the proposal by documenting the failure. Toward Understanding Generative Data Augmentation derives stability bounds showing that synthetic data helps most when $m$ is small. It also reports that GAN data reduced ResNet18 on CIFAR-10 from 94.4% to about 92.6%, while diffusion data (EDM) raised it to 97.5%. Synthesizing Diverse Lung Nodules Wherever Massively observes a dip when the amount of synthetic data goes from 1× to 3×. That paper attributes the dip to the detector overfitting generator artifacts, which is exactly what our mapper is meant to absorb.

Other papers supply the mechanism, and they also mark how we differ:

  • AADA (Mind the Discriminability) freezes source features and pulls target features into them. SVD analysis shows that symmetric DANN collapses feature diversity.
  • AsyFOD (AsyFOD) applies stop-gradient to sparse target features in few-shot detection.
  • Rad et al. (Feature Mapping for 3D Pose) learns a residual network that maps real features into a frozen space learned from rendered CGI.

All three anchor across two genuinely different deployment domains. None of them treats a deep generator's output as the domain to be adapted. Rad et al. anchor the synthetic side, which made sense because its rendered labels were perfect. With learned generators the reliability ordering is usually the opposite, and testing that reversal is the point of this proposal.

Two closer papers act on the generator or on stored data, not on downstream training:

Our method works with any frozen generator, requires no unrolled optimization, and costs one small extra network during training only.

Single-Shot Domain Adaptation via Target-Aware Generative Augmentation (SiSTA) exports synthetic target data offline and adapts with neighborhood consistency, with no feature anchor. Our method could be dropped into its adaptation stage.

One finding constrains us. Data Efficient Unsupervised Domain Adaptation for Cross-Modality Image Segmentation shows that anchoring breaks down at extreme sparsity, with manifold smearing when only three 2D slices are available. This is why we predict a crossover rather than universal dominance.

Research Program

The work has three parts.

The first is a controlled study on CIFAR-10. We train ResNet18 under standard augmentation while varying the number of real images per class over roughly 1% to 100% of the training set. The synthetic data comes from three sources. Two are GAN sources (cDCGAN and StyleGAN2-ADA), which are known to hurt. The third is EDM, which is known to help. At every point we compare four directions:

  • naive mixing;
  • a symmetric mapper, identical to ours but with the stop-gradient removed;
  • our synthetic-to-real asymmetric mapper;
  • the reverse real-to-synthetic mapper in the style of Rad et al.

We will also repeat the SVD spectrum analysis from AADA on the real features learned by $f$. This tests the mechanism directly: naive GAN mixing should narrow the spectrum of real features, and our method should not.

The second part is a medical transfer to 3D segmentation on MSD-Pancreas, using Difftumor synthetic tumors (Optimizing Synthetic Data for Enhanced Pancreatic Tumor Segmentation). Here $g$ acts on decoder features, and the prototypes are computed per class within tumor and pancreas masks. This tests whether the idea survives dense prediction and a realistic generator.

The third part is a stress test using the paper's elastic boundary-noise setting. We state the limitation up front: boundary noise corrupts the labels, not only the features, so a feature mapper can only partly absorb it. We report whatever we find.

Baselines

Do-nothing reference. We train on real data with standard augmentation only. Zheng et al. report 94.4% for ResNet18 on CIFAR-10 in this setting. Pancreas results are reported per backbone; SwinUNETR is the main one because Difftumor gives it the largest reported gain:

Strongest existing method. This is naive mixing with the best available generator, using the same real data and synthetic pool. Reported numbers:

For the "harmful generator" condition the target is about 92.6%, the GAN-mixed figure. We reproduce all of these numbers in our own pipeline before comparing, so that differences reflect the method and not the codebase. Every arm uses identical real and synthetic data, epochs, optimizer, and the same hyperparameter search budget of 20 trials over $\gamma$, $\lambda$ and the learning rate. Results are averaged over five seeds.

What counts as beating them. Our method must meet two conditions:

  • With GAN data, it must reach at least the 94.4% real-only reference, so that the liability is fully removed.
  • With EDM data and pancreas data, it must not fall below naive mixing, and should exceed it by more than seed variance.

Ablations of the key ingredient. The symmetric mapper, which differs only by the missing stop-gradient, isolates the asymmetry. Removing $g$ while keeping the prototype loss isolates the mapper. The real-to-synthetic mapper tests the direction.

Why not just…?

Why not just use a diffusion model and skip all of this? This is the most serious threat. Zheng et al. show EDM data already helps on CIFAR-10. If our gains disappear whenever the generator is good, the method is a patch for weak generators. The EDM arm settles this. We expect a small gain at most there, and a large effect only with GANs and at high data-to-generator-quality ratios. The pancreas experiment matters because even a strong 3D latent diffusion generator leaves a large gap to real data in that domain.

Why not just fix the generator with feature alignment, as in Feature-Aligned Diffusion? That approach requires fine-tuning the generator for every expert and dataset. It needs a good expert, which presupposes plenty of labeled real data, and its authors report reduced variability in some classes. We include it as an arm on CIFAR-10 using the public code. The two approaches are complementary: if aligned-generator data plus our mapper beats either alone, the methods stack.

Why not just down-weight the synthetic data, give it separate batch normalization, or pretrain on synthetic data and fine-tune on real? All three are cheap and plausible, and separate batch statistics are in effect a crude mapper. We run each under the same tuning budget. If one of them matches our method, especially separate batch normalization, then the contribution reduces to "isolate synthetic statistics." We will report that finding plainly rather than claim the mapper is necessary.

Proposed Experiments and Decisive Checks

The first decisive check is on CIFAR-10 with GAN data. Our method should recover at least 94.4%. The symmetric-mapper ablation should not, which shows that the stop-gradient is what matters.

The second check is the crossover. Plotting accuracy against $m$, the synthetic-to-real direction should beat the real-to-synthetic direction at large $m$, and the order should reverse at the smallest $m$. If no crossover appears, the anchoring-direction thesis is wrong in its stated form.

The third check is the SVD spectrum of real features. It should be preserved under our method and narrowed under naive GAN mixing, which would confirm the mechanism borrowed from AADA.

The fourth check is on MSD-Pancreas. SwinUNETR with our method should exceed the 56.10% DSC Difftumor reference by more than seed variance. In the boundary-noise condition we report how much of the drop from 48.48% to 32.49% is recovered, with no preset claim.