Image generators such as Stable Diffusion can produce training images of styles a dataset never contained, such as sketches, sculptures, or paintings of each class. Adding them to real training data makes classifiers far more robust to those shifts. The catch is that the benefit does not keep growing as more synthetic images are added. Classifiers can tell generated images apart from real ones, and theory predicts that piling on off-distribution data eventually drags the model away from what it should learn. This note proposes a simple, testable fix. Never let the classifier see a purely synthetic image. Instead, paste a patch from a generated image into a real image, or the reverse, using the standard CutMix recipe (cut a rectangle from one image, paste it into another, and mix the labels by area). Every training example then contains real pixels, so "is this image generated?" stops being a useful shortcut.
The specific claim is that pairing each synthetic image with a real partner, rather than mixing images drawn at random from a pooled dataset, moves the point where adding more synthetic data starts to hurt toward much larger synthetic-to-real ratios. It should do so without erasing the out-of-distribution styles that made the synthetic data useful. The prediction is sharp because the two pairing schemes coincide when synthetic data is scarce and diverge when it dominates. At a ratio of 8 synthetic images per real image, random pairing makes about 80% of pairs synthetic–synthetic, while forced pairing makes none.
Why this question, and what prior work leaves open
Three results motivate the proposal. Leaving Reality to Imagination trains classifiers on a 1:1 mix of real and Stable Diffusion images generated from 80 style templates and reports roughly 15% absolute gains on ImageNet-Sketch and ImageNet-R. It tests only that one ratio. Let Synthetic Data Shine (DRSF) shows that generated images form their own cluster in feature space, which it calls the "synthetic data paradox." The Value of Out-of-Distribution Data proves and observes empirically a non-monotonic curve: a little off-distribution data lowers error, but large amounts raise it sharply. That paper also reports that standard augmentation does not prevent the rise unless samples are explicitly reweighted. Taken together, these papers suggest that the useful operating range of synthetic data is set by how separable synthetic and real images are to the classifier.
Two results constrain the design. First, the "fine-tuning paradox" in Leaving Reality to Imagination shows that pulling the generator toward the real distribution destroys the robustness gain. Any anchoring scheme that blends synthetic content back toward real-looking content risks the same failure. This is why the core mechanism is patch pasting, which leaves a sketch patch looking like a sketch, rather than pixel averaging (Mixup), which fades the style. Second, A Generalized Theory of Mixup proves that interpolation weights restricted to [0,1] shrink data variance. Pixel-averaging real with synthetic images should therefore contract exactly the stylistic spread we want, and we treat Mixup as a variant the theory predicts will do worse.
Several papers supply methods. Cut to the Mix shows that crude CutMix, even when it creates anatomically impossible images, beats elaborate realism-preserving augmentation on small CT datasets. It improves macro Dice by 4.9 on AMOS, against 2.0 for CarveMix. That result licenses using unrealistic composites. The generators come from Leaving Reality to Imagination for ImageNet-scale work and from CDGA, described in Leveraging Generative Foundation Models for Domain Generalization, for multi-domain benchmarks.
The closest existing work combines the same two ingredients but differs in important ways. DistDiff applies Mixup after generation as an independent add-on, never asks how real and synthetic samples are paired, and deliberately keeps generation in-distribution. DRSF does mix real and synthetic features. However, it does so in feature space, with scalar interpolation and a gradient-reversal adversary, for weather-shift detection with prompted domains, and it never studies the synthetic ratio. FIXED and Decoupled Mixup mix only observed real images. None of these papers treats the pairing rule between data sources as the variable, and none measures how it reshapes the curve of error against synthetic ratio.
Core Claim
Let $r = m/n$ be the number of synthetic images $m$ per real image $n$. We predict three things. First, the ratio $r^*$ that minimizes out-of-distribution error is larger under cross-source CutMix than under random-pool CutMix or no mixing. Second, the gap between the two pairing schemes is near zero for $r \le 1$ and grows with $r$. Third, the size of the gain tracks a measurable quantity: how accurately a linear probe on the classifier's penultimate features can tell real from synthetic inputs. If the probe accuracy does not fall, or the fall does not track the out-of-distribution gain, then the "no synthetic-only shortcut" explanation is wrong, even if accuracy improves.
Research Program
We first generate a fixed synthetic pool with frozen Stable Diffusion, using class names and the 80 style templates from Leaving Reality to Imagination, for the 100 classes of ImageNet-100 (the subset used by Real-Fake). The pool is large enough to reach $r = 16$. For the domain generalization benchmarks, we generate CDGA-PG-style prompted images.
Training then varies two factors: the ratio $r$, over 0, 0.25, 0.5, 1, 2, 4, 8, and 16, and the mixing rule. The mixing rules are none; random-pool CutMix; cross-source CutMix, in which every synthetic image is pasted into or receives a patch from a real image and the leftover real images are mixed among themselves; and the same two pairing rules with Mixup. Labels follow patch area as in standard CutMix.
Every run uses the same architecture (ResNet-50), the same number of gradient steps, the same image count per batch, and the same hyperparameter search budget. This ensures that a larger $r$ means more diverse data, not more compute. After training, we fit the real-versus-synthetic linear probe on held-out images. As a check on the variance argument, we also compute the effective rank of penultimate features on ImageNet-R, following Training-free Design of Augmentations, to test whether Mixup contracts the representation of shifted styles more than CutMix does.
Baselines
The naive reference point is a real-only classifier trained with standard crop-and-flip augmentation. The basic synthetic reference is the fixed 1:1 real-plus-synthetic recipe of Leaving Reality to Imagination, with no mixing. That paper reports about 15% absolute gains on ImageNet-Sketch and ImageNet-R, but the file gives no absolute accuracies, so we reproduce it on our class subset rather than borrow a number.
The strongest competitor on the multi-domain benchmarks is plain ERM (standard training that minimizes average loss) trained on CDGA-augmented data, from Leveraging Generative Foundation Models for Domain Generalization. It reports 68.4 average accuracy on PACS, against 64.3 for ERM, with training-domain validation for model selection. It is the strongest because it already beats specialized domain-generalization losses using generated data alone. We will reproduce it inside DomainBed with the same selection protocol and add our mixing on top. Beating it means a gain over our reproduced CDGA number that exceeds seed variance across at least three seeds.
The baseline that targets the shortcut problem directly is DRSF-style adversarial alignment (Let Synthetic Data Shine). We implement it as a gradient-reversal real-versus-synthetic discriminator plus feature-level soft fusion, under the same step and tuning budget.
The baseline for the ratio problem is the α-weighted objective of The Value of Out-of-Distribution Data, which down-weights synthetic loss. We tune α per ratio on the same validation split.
The ablation that isolates the key ingredient is random-pool CutMix at matched ratio, which differs from our method only in the pairing rule. A second ablation, cross-source Mixup, isolates patch pasting from pixel blending.
Why not just…?
Why not just tune the synthetic ratio and stop at the best one? This is the cheapest alternative, and it may capture much of the gain if the error minimum sits near $r = 1$. Our sweep reports the best accuracy over ratios for every method, so the comparison is "best tuned ratio versus best tuned ratio." If cross-source CutMix does not raise that best accuracy, and only moves $r^*$ to the right, the contribution reduces to a statement about data efficiency. We would report that honestly.
Why not just reweight synthetic samples, as the out-of-distribution data paper does? The α-weighting in The Value of Out-of-Distribution Data reduces the influence of synthetic images, but it does nothing to stop the classifier from learning that generated images are distinguishable. It may therefore discard useful synthetic signal along with the harmful part. This is a real threat, and it is run as a tuned baseline at every ratio. The probe measurement separates the two mechanisms: reweighting should leave real-versus-synthetic separability high, while cross-source mixing should lower it.
Why not just apply CutMix to the pooled dataset, as DistDiff treats Mixup? At low ratios this is nearly identical to our method, and we expect it to tie. Our claim lives entirely at high ratios, where random pairing produces mostly synthetic–synthetic composites that still carry the synthetic signature. If random-pool CutMix matches cross-source CutMix at $r \ge 4$, the pairing rule does not matter and the idea is falsified.
Why not just align features adversarially, as DRSF does? Gradient-reversal training adds a minimax objective and extra tuning. It also aligns whole feature distributions, which can wash out the style variation that drives robustness, the same failure the fine-tuning paradox exposes on the generator side. It is a direct baseline here. If it matches our method at equal tuning budget, our method's only advantage is simplicity.
Proposed Experiments
The primary benchmark is ImageNet-100 training, evaluated on the matching classes of ImageNet-Sketch, ImageNet-R, and ImageNet-V2, plus clean validation accuracy. The second benchmark is DomainBed PACS, OfficeHome, and DomainNet, leaving one domain out at a time with CDGA-generated images. No reported result was found on Lacuna for absolute ImageNet-Sketch accuracy under the 1:1 synthetic recipe, so all ImageNet-side baselines are reproduced. For PACS, the anchor is the 68.4 reported by CDGA.
The decisive checks are as follows. First, plot out-of-distribution error against $r$ for every mixing rule. The method succeeds if cross-source CutMix keeps improving or stays flat at ratios where the no-mixing and random-pool curves turn upward. Second, at $r \ge 4$, cross-source CutMix should beat random-pool CutMix by more than seed variance on ImageNet-Sketch and ImageNet-R. Third, cross-source Mixup should underperform cross-source CutMix on the style-shift test sets, and feature effective rank on ImageNet-R should be lower under Mixup, as the variance argument predicts. Fourth, across all runs, the fall in real-versus-synthetic probe accuracy should correlate with out-of-distribution gain. Finally, clean ImageNet-100 accuracy must not drop by more than one point; a robustness gain bought with in-distribution loss would be reported as a trade-off rather than a win.
References
- Leaving Reality to Imagination: Robust Classification via Generated Datasets
- Let Synthetic Data Shine: Domain Reassembly and Soft-Fusion for Single Domain Generalization
- The Value of Out-of-Distribution Data
- A Generalized Theory of Mixup for Structure-Preserving Synthetic Data
- Cut to the Mix: Simple Data Augmentation Outperforms Elaborate Ones in Limited Organ Segmentation Datasets
- Leveraging Generative Foundation Models for Domain Generalization
- Distribution-Aware Data Expansion with Diffusion Models
- FIXED: Frustratingly Easy Domain Generalization with Mixup
- Decoupled Mixup for Out-of-Distribution Visual Recognition
- Real-Fake: Effective Training Data Synthesis Through Distribution Matching
- Training-free Design of Augmentations with Data-centric Principles