Models trained on simulated or rendered data, such as video-game street scenes, CAD renderings of objects, or simulated driving, often fail when deployed on real cameras. Language-model research has built a toolkit for deciding which training examples are worth keeping. Its main idea is to score each example by how surprised a small reference model is by it, where surprise is measured as the model's cross-entropy loss (perplexity is the exponential of this loss), and then to keep the informative ones. Sim-to-real pipelines almost never do this. They either train on every synthetic sample or mix synthetic and real data at a fixed ratio. The reason a naive transplant fails is instructive. In a simulator, a synthetic sample can be hard for two opposite reasons. It may show real, physically meaningful difficulty, such as an odd pose, heavy occlusion, or a rare hazard. Or it may be simulator garbage, such as a rendering bug, an impossible configuration, or an annotation glitch. Simulators also contain the reverse problem: samples that look easy only because the model has found a simulator shortcut, like a texture statistic that happens to predict the label.
This note proposes to score each synthetic sample by its loss under two small reference models, one trained on synthetic data and one trained on a small labeled slice of real data, and to select samples by the difference between the two losses. A sample the synthetic-trained model finds hard but the real-trained model finds predictable carries difficulty that the real world supports. A sample that is hard for both is probably noise. A sample that is easy for the synthetic model but hard for the real one is probably a shortcut. The core claim is that this two-world loss contrast selects synthetic subsets that transfer better to real test data than raw-loss selection, realism filtering, or using all the data. It should also do this at a scoring cost of two cheap forward passes per sample.
Where This Sits in Prior Work
The motivating evidence comes from language-model data selection. Perplexed by Perplexity prunes pretraining text using a 125M reference model's perplexity. It finds that keeping high-perplexity data works best on curated text (The Pile), that medium perplexity works best on noisy web text (Dolma), and that keeping low-perplexity data is worst everywhere. That split already suggests raw loss mixes signal with noise, and the right cut depends on how noisy the source is. Mixture-of-Skills argues directly that raw perplexity confuses intrinsic complexity with learning value. Efficiently Selecting Response Generation Strategies by Self-Aligned Perplexity shows that on teacher-generated synthetic text, the instruction-following difficulty ratio correlates with downstream gains worse than raw perplexity does, because teacher style differs from student style. We read that paper as evidence that difficulty scores must be anchored to the target distribution. Its own anchor, a pool of the student's correct zero-shot answers, operates on whole generation strategies and has no visual counterpart.
The method comes most directly from two papers that contrast losses across a pair of models: Preference Curriculum and the four-quadrant scheme of FRAME. Both contrast a small and a large model to separate unlearnable data from data that needs more capacity. We keep that quadrant logic but swap the axis: the two references differ in which world they were trained on, not in size. The ratio form of Cherry-LLM's instruction-following difficulty is the same kind of move. It compares losses under two conditionings to cancel out intrinsic complexity. We compare two training distributions instead. From Superfiltering, which reports rank correlation above 0.8 between difficulty scores from a 124M model and from 7B–13B models, we take the hypothesis that small references can score data for large targets. We test whether that stability survives a domain gap.
On the sim-to-real side, the closest selection work is RLS3. It has a reinforcement-learning agent arrange Unity scenes that make a vision-language model fail, and it reports about 40% better data efficiency than random scenes. It rewards failure within the simulator and was never tested on real data, so by construction it would chase simulator-only failures as readily as transferable ones. Diffusion Curriculum schedules a synthetic-to-real spectrum through a diffusion guidance weight and validation confidence, not through per-sample loss. Curriculum Domain Adaptation builds an easy-to-hard curriculum from hand-designed urban-scene priors. Automated Synthetic-to-Real Generalization (ASG) keeps the model close to a frozen ImageNet network on synthetic inputs but treats every synthetic image identically. CoC-VLA feeds all simulated CARLA edge cases to a teacher. None of these asks, sample by sample, whether a synthetic example's difficulty is endorsed by the real world. The simulation literature explains why this question matters. Jet taggers trained on simulation learn artifacts of the simulation code (Bauer et al., Kagan et al.), which is exactly the failure a single-reference difficulty score cannot detect.
Core Claim
For a synthetic sample $x$ with label $y$, let $\ell_S(x,y)$ be the loss of a small reference model trained only on synthetic data. Let $\ell_R(x,y)$ be the loss, on the same synthetic input, of a small reference model of the same architecture trained only on a small labeled real slice. Both start from the same ImageNet initialization. We define the transfer score
$$s(x,y) = \ell_S(x,y) - \ell_R(x,y),$$
and keep samples with high $s$, subject to a noise gate $\ell_R(x,y) < \tau$. The threshold $\tau$ is set at a percentile of $\ell_R$ on held-out real validation data.
The four regions of the $(\ell_S,\ell_R)$ plane have predicted meanings. Low $\ell_S$ with low $\ell_R$ means redundant. High $\ell_S$ with low $\ell_R$ means transferable difficulty. Low $\ell_S$ with high $\ell_R$ means a simulator shortcut. High values of both mean unlearnable noise. The claim is falsifiable in two ways. First, these regions should line up with planted corruptions of known type. Second, a target model trained on the high-$s$ subset should beat a model trained on the same number of samples chosen by raw $\ell_S$. For autoregressive outputs such as driving rationales, $\ell$ is the mean token loss on the rationale given the image and question. This makes the score a literal perplexity contrast.
Research Program
The first stage builds and validates the score where ground truth is controllable. On VisDA-17 synthetic renderings, we plant two corruptions in disjoint 5% subsets. The shortcut corruption is a class-correlated rendering cue, a faint per-class texture overlay. The noise corruption is randomized labels paired with degraded renders. We then measure how well the chosen region of the $(\ell_S,\ell_R)$ plane recovers each planted set. Raw high-loss selection should pull in the noise set. Raw low-loss selection should retain the shortcut set. The contrast score should reject both.
The second stage tests transfer on two standard sim-to-real benchmarks: VisDA-17 classification, and GTA5→Cityscapes and SYNTHIA→Cityscapes segmentation, where the loss is the per-image mean pixel cross-entropy. We sweep the kept fraction from 10% to 100% of the synthetic pool and the real-slice size from 1% to 10% of real training labels. The real-slice sweep matters because the real reference is weak when the slice is small. Finding the smallest slice at which the score still helps is a result in itself.
The third stage tests weak-to-strong scoring across the gap. ResNet-18 references score data for ResNet-101 and DeepLab-class targets. We report Spearman correlation between the small-reference and large-reference scores, following Superfiltering's protocol.
The fourth stage extends to language outputs, CARLA-style simulated driving question-answering transferred to nuScenes-VLM. Small vision-language references score simulated rationales, and a LLaVA-v1.5-class target is fine-tuned on the selected subset. This stage depends on whether CoC-VLA's CARLA-VLM split is obtainable. If it is not, we generate CARLA scenarios ourselves and describe them as a new, non-comparable setting. STSBench (ivc-lrp/STSBench) supplies real nuScenes scenario questions for evaluation.
Baselines
The naive reference point is training the target on all synthetic data plus the same small real slice, the latter either mixed in at a fixed ratio or used for a short final fine-tune. We run both variants and treat the better one as the do-nothing baseline. Every selection method gets the same real slice, so any gain comes from choosing synthetic data and not from having extra real labels.
The first selection baseline is raw-loss selection under the synthetic reference, keeping high, medium, or low $\ell_S$. This is the direct transplant of Perplexed by Perplexity. It is the most important competitor because it isolates our key ingredient, the real reference. We report the best of its three cuts at each budget, which is generous to the baseline.
The second is realism selection. A binary classifier trained to separate synthetic from real images at equal cost ranks synthetic samples by how real they look. This is the obvious domain-adaptation-flavored filter, and it tests whether "looks real" already captures "transfers."
The third is the two single-reference ablations: selecting by low $\ell_R$ alone, and removing the noise gate. These show whether the difference, and not either term alone, drives the gains.
The strongest published systems are reference points and are rerun where feasible. On GTA5→Cityscapes, ASG reports 31.47% mIoU with FCN-VGG16 (Automated Synthetic-to-Real Generalization). On VisDA-17, ASG reports 61.1% with ResNet-101, against 58.7% for fixed proxy guidance. On SYNTHIA→Cityscapes, Curriculum Domain Adaptation reports 29.0% mIoU on 16 classes (Curriculum Domain Adaptation). Self-ensembling reports 92.8% on VisDA-2017 using unlabeled target data and engineered augmentation (Self-ensembling for visual domain adaptation). On nuScenes-VLM, CoC-VLA reports CIDEr 23.21, displacement error 1.44, and collision rate 1.58% (CoC-VLA).
These published numbers come from regimes that differ from ours. ASG and Curriculum Domain Adaptation use no real labels, and self-ensembling uses unlabeled target data. They therefore bound the landscape but are not head-to-head comparisons. The fair tests are the reruns. We apply ASG's proxy guidance, and separately self-ensembling, with and without our selected subset, on the same backbone, the same real slice, and the same number of training steps. Each method gets the same hyperparameter search budget of 16 configurations tuned on real validation data. Seeds are fixed, and we report three runs.
Beating a baseline means a gain larger than seed variance at a matched kept fraction and budget. The decisive result is that the high-$s$ subset outperforms the best raw-loss cut and the realism filter at equal data size, and that adding it to ASG or self-ensembling improves them rather than being absorbed.
Why not just…?
Why not just generate more synthetic data? Language-model results show that curated subsets beat full sets: 5% of Alpaca chosen by instruction-following difficulty won 49 to 26 against the full set in human evaluation (Cherry-LLM). The jet-tagging work suggests that more simulation data mostly reinforces simulator artifacts. Neither result is a sim-to-real image experiment, so our kept-fraction sweep settles the question directly. If the 100% point matches or beats every selected subset on real test data, the premise fails.
Why not just select by the synthetic model's loss, as in text pruning? A single reference cannot tell physical difficulty from simulator bugs, because both raise its loss. Nor can it tell genuine ease from a shortcut, because both lower it. The planted-corruption experiment tests this claim directly: raw high-loss selection should keep the noise set, and raw low-loss selection should keep the shortcut set. If raw medium-loss selection matches our score on real transfer, as it did on noisy text in Perplexed by Perplexity, the real reference adds nothing and we would report that.
Why not just keep synthetic images that look most realistic? Realism is a property of the image, while transfer is a property of the relationship between image and label. A photorealistic render with a label that depends on a simulator quirk passes a realism filter but should show high $\ell_R$. This alternative is a real threat on benchmarks where the dominant gap is appearance, and the realism baseline is included to measure exactly how much of our gain it already captures.
Why not just fine-tune on the small real slice and ignore selection? This is the most serious threat. A final real fine-tune may erase whatever differences synthetic selection makes. The do-nothing baseline includes this variant. We also report results as the real slice grows, since we expect selection to matter most when real labels are scarce and the gap to shrink as real labels increase. If selection gives no gain even at a 1% real slice, the idea should be dropped.
Decisive Checks
Three results would establish the claim. First, the four regions of the $(\ell_S,\ell_R)$ plane recover the planted shortcut and noise sets with clearly higher precision than either single loss. Second, at matched kept fractions on VisDA-17 and GTA5→Cityscapes, the high-$s$ subset beats the best raw-loss cut and the realism filter on real test accuracy or mIoU, and the gain persists when stacked with ASG or self-ensembling. Third, small-reference scores keep a high rank correlation with large-reference scores across the domain gap, so the method stays cheap.
A clean negative result on the second check, with the first check succeeding, would also be informative. It would mean that sample-level diagnosis of simulator artifacts is possible but does not by itself change transfer, which would redirect effort toward fixing the simulator rather than filtering its output.
References
- Perplexed by Perplexity: Perplexity-Based Data Pruning With Small Reference Models
- Mixture-of-Skills: Learning to Optimize Data Usage for Fine-Tuning Large Language Models
- Efficiently Selecting Response Generation Strategies for Synthetic Data Construction by Self-Aligned Perplexity
- Preference Curriculum: LLMs Should Always Be Pretrained on Their Preferred Data
- FRAME: Boosting LLMs with A Four-Quadrant Multi-Stage Pretraining Strategy
- From Quantity to Quality: Boosting LLM Performance with Self-Guided Data Selection for Instruction Tuning
- Superfiltering: Weak-to-Strong Data Filtering for Fast Instruction-Tuning
- RLS3: RL-Based Synthetic Sample Selection to Enhance Spatial Reasoning in Vision-Language Models for Indoor Autonomous Perception
- Diffusion Curriculum: Synthetic-to-Real Generative Curriculum Learning via Image-Guided Diffusion
- Curriculum Domain Adaptation for Semantic Segmentation of Urban Scenes
- Automated Synthetic-to-Real Generalization
- CoC-VLA: Delving into Adversarial Domain Transfer for Explainable Autonomous Driving via Chain-of-Causality Visual-Language-Action Model
- Jet substructure classification in high-energy physics with deep neural networks
- Boosted Jet Tagging with Jet-Images and Deep Neural Networks
- Self-ensembling for visual domain adaptation