Models trained on rendered images, such as CAD renders of objects or frames from the video game Grand Theft Auto V, usually fail on real photographs. On the VisDA benchmark, a network scores 99% on synthetic images and about 28% on real ones (VisDA). Almost every fix treats the synthetic pool as a single block. Domain adaptation methods align the two image distributions as wholes, and Automated Synthetic-to-Real Generalization regularizes the whole training run toward an ImageNet-pretrained copy of the network. The synthetic pool is not uniform, though. Some renders teach the shapes and parts that carry over to real photos. Others mainly teach simulator quirks, such as sharp edges, fixed lighting and canonical poses, that the network can exploit as shortcuts.
This note proposes to score each synthetic image by one question: does training on it move the model in the same direction that real target images would? We measure this as the agreement between the image's gradient and a gradient computed on unlabeled real images, using labels from a separate, slowly updated teacher model. The score then reweights the supervised synthetic loss inside an otherwise standard adaptation pipeline. The proposal is mostly a careful test of whether this signal works when no real labels exist, and whether it measures something beyond "looks like real data" or "matches the real class mix."
Core Claim
Gradient-agreement data selection already works when a labeled target development set exists. Optimizing Data Usage via Differentiable Rewards (DDS) scores training examples by the cosine between their gradient and the gradient on a labeled dev set. OptimSyn uses the same gradient dot product as a reward for generating synthetic LLM training data. OptimSyn also reports that embedding-space closeness to real data is a misleading proxy for usefulness, which it calls the "looks good" trap. Neither method has been applied to simulator-to-real vision. Both need target labels, and real images in sim-to-real settings are unlabeled.
The obvious replacement is to pseudo-label the real images with the model being trained. This is circular. The model's errors on real images come partly from simulator shortcuts, so synthetic images whose gradients agree with those errors would be up-weighted and the shortcuts reinforced.
Our claim has three parts. First, computing the target gradient from pseudo-labels of a decoupled teacher, an exponential moving average of the student restricted to confident target images, breaks most of this circularity. Second, the resulting per-image weights recover a substantial fraction of what an oracle labeled dev set would give. Third, the gain is not explained by embedding proximity or by matching the target class distribution, and it adds on top of a strong pseudo-labeling adaptation method rather than replacing it.
For synthetic image $i$ at training step $t$, let $g_i$ be the gradient of its supervised loss with respect to the final block of the network. Let $\bar g_T$ be the mean gradient, over a cached batch of real target images, of the loss against the teacher's confident pseudo-labels. The score and weight are
$$ s_i = \cos(g_i, \bar g_T), \qquad w_i = \frac{\max(s_i, 0) + \epsilon}{\frac{1}{B}\sum_{j}\left(\max(s_j,0)+\epsilon\right)}, $$
where $B$ is the batch size and $\epsilon$ is a floor that keeps every image's weight above zero. The floor and the normalization are deliberate. On the Difficulty of Learning a Meta-network for Training Data Selection proves that learned selectors collapse into a "winner-takes-most" regime unless batches are very large ($N=1024$, up to 180GB of memory). We avoid training a meta-network altogether. The score is computed directly from the model's current state, so no selector parameters exist to collapse, and the floor bounds how far any single image can dominate. Using only final-block gradients, together with the first-order trick from DDS, keeps the cost close to DDS's reported 1.5x to 2x overhead.
Research Program
The work has three parts.
Build the method. We implement the weighting inside two pipelines. The first is plain source training with the weighted synthetic loss. The second is a mean-teacher pseudo-labeling adaptation method of the kind that won VisDA classification. In the second, our weights affect only the supervised synthetic term.
Map the circularity. We compute the target gradient three ways: from the student's own pseudo-labels, from the decoupled teacher, and from true labels on VisDA's labeled COCO validation split, which serves as an oracle. We track how the weights produced by each source diverge over training.
Find out what the weights encode. We measure how much of the variance in the learned weights is explained by class identity, by embedding distance to the target, and by render attributes such as viewpoint and lighting where the simulator exposes them.
This program uses the literature in three roles. DDS and OptimSyn supply the scoring signal. The meta-network difficulty paper constrains the design, which is why there is no learned selector and the batch can stay small. Reinforced Training Data Selection for Domain Adaptation motivates the work: it showed in NLP that training on 40 to 50% of the source data, chosen by an unlabeled-target discrepancy reward, beats training on all of it. That method selects coarse bags of examples using distribution distance, which is exactly the proxy OptimSyn calls into question. Bayesian Active Learning for Sim-to-Real Robotic Perception chooses which real images to send for labeling using hand-built uncertainty rules. We instead weight the synthetic pool with no labeling budget at all. Meta-Sim changes the scene generator, and RLS3 produces hard synthetic scenes but never tests on real images. Our method selects from a fixed pool and is evaluated only on real data.
Baselines
Source-only training. This is the do-nothing reference: uniform weights, same backbone, same schedule. The VisDA paper reports about 28% on real COCO images for AlexNet. Automated Synthetic-to-Real Generalization reports 28.2% on VisDA-17 classification. We rerun our own source-only number on our backbone rather than borrowing either figure.
Proxy guidance (ASG) without selection. This is the strongest existing method that processes all synthetic data uniformly. ASG reports 58.7% on VisDA-17 with the ImageNet KL penalty alone, 61.1% with its reinforcement-learned layer-wise learning rates, and an improvement from 22.17% to 29.64% mIoU for FCN-Res50 on GTA5-to-Cityscapes. We run ASG's KL penalty with and without our weights. Our weights must add at least two points on top of proxy guidance, with the same backbone and the same number of epochs, to count as complementary.
Strongest adaptation pipeline. We use a mean-teacher pseudo-labeling method. The VisDA challenge winner reached 92.8% classification and 47.5 mIoU on segmentation with multi-stage pipelines; ASG used as pretraining for adaptation reached 84.6%. We will not reproduce the winner's ensemble. Instead we run one reproducible mean-teacher configuration with a fixed hyperparameter budget, shared with every variant. Beating it means a consistent gain across three seeds when only the synthetic weights change.
Unlabeled-target selection by distribution distance. These are the two heuristics the claim targets. The first selects synthetic images by nearest-neighbor distance to real image embeddings. The second weights them to minimize the distance between synthetic and real feature distributions, as the MMD reward in the reinforced selection paper does. Both get the same compute and tuning budget as our method.
Class-prior reweighting. Synthetic samples are reweighted so that the class distribution matches the teacher's predicted class distribution on the real target.
Ablations. Two ablations isolate the key ingredient. The first swaps the decoupled teacher for the student's own pseudo-labels. The second swaps it for oracle COCO validation labels, which act as the DDS upper bound. A third ablation removes the floor $\epsilon$ to test whether collapse returns at small batch sizes.
Why not just…?
Why not just run DDS with pseudo-labeled real images? This is the closest alternative. It differs from our method only in where the pseudo-labels come from, so the student-pseudo-label ablation tests it directly. If self-pseudo-labels match the decoupled teacher across training, our contribution reduces to "DDS transfers to sim-to-real with pseudo-labels." That is still a useful, untested result, but the circularity claim would be withdrawn. We expect the two to diverge late in training, when the student's real-image errors have stabilized around shortcuts.
Why not just match the target class distribution? Much of the apparent value of selection under domain shift may simply be correcting for different class frequencies. That correction is cheap. We run class-prior reweighting as a baseline and decompose our weights by class. If class identity explains most of the weight variance, or class-prior reweighting matches our accuracy, the per-image signal is not doing real work and we will report that.
Why not just keep synthetic images that look most like real ones? OptimSyn found that embedding proximity misleads in text, but no paper in our file tests this for rendered images. The comparison is settled with the selection baselines above and with the utility-correlation check described below. It is possible that, in vision, looking real and being useful are well correlated. Finding that would be informative in itself.
Why not just use a strong adaptation method and skip selection? Pseudo-labeling pipelines already close most of the gap (92.8% in the VisDA challenge), so selection must earn its place inside them. This is why the main experiment applies our weights within mean-teacher training rather than comparing against it.
Why not just use uncertainty sampling? Deep Active Learning with Adaptive Acquisition reports that uncertainty sampling chases outliers and noise. On a synthetic pool, the images the model is most uncertain about may well be rendering failures. We include a high-loss weighting baseline for completeness, but we do not expect it to be competitive.
Proposed Experiments
Benchmarks. The primary benchmark is VisDA-C. The 152,000 synthetic CAD renders are the training pool. The COCO validation images serve as unlabeled target data, and their labels are used only for the oracle. Results are reported on the YouTube Bounding Boxes test frames. Because the test set differs from the COCO split, any overfitting of the oracle to its dev set will be visible. The secondary benchmark is GTA5-to-Cityscapes segmentation, where per-pixel gradients are pooled to one score per image and results are reported in mean intersection-over-union. Where the simulator exposes viewpoint and lighting, we analyze the BlenderProc-to-EDAN robotics setting from the Bayesian active learning paper.
First decisive check: does the score predict utility? On a 5,000-image VisDA subset, we measure the true utility of 200 random 100-image groups by retraining with each group removed and recording the change in real test accuracy. We then compute the rank correlation between this measured utility and three scores: gradient agreement, embedding proximity, and the MMD contribution. The proposal survives this check only if gradient agreement correlates with utility clearly better than both distribution-distance scores.
Second decisive check: does decoupling the teacher matter? We compare self-pseudo-labels, the decoupled teacher, and the oracle on final accuracy and on the overlap of their top-weighted images over training.
Third decisive check: does selection add to adaptation? We measure the gain of weighted over uniform synthetic loss inside the mean-teacher pipeline, across three seeds.
Interpreting the results. If the third check shows no gain while the first two succeed, the honest conclusion is that per-image selection helps weak pipelines but is absorbed by strong pseudo-labeling. That would still tell practitioners when curating a synthetic pool is worth the effort.
References
- VisDA: The Visual Domain Adaptation Challenge
- Automated Synthetic-to-Real Generalization
- Optimizing Data Usage via Differentiable Rewards
- OptimSyn: Influence-Guided Rubrics Optimization for Synthetic Data Generation
- On the Difficulty of Learning a Meta-network for Training Data Selection
- Reinforced Training Data Selection for Domain Adaptation
- Bayesian Active Learning for Sim-to-Real Robotic Perception
- Meta-Sim: Learning to Generate Synthetic Datasets
- RLS3: RL-Based Synthetic Sample Selection to Enhance Spatial Reasoning in Vision-Language Models for Indoor Autonomous Perception
- Deep Active Learning with Adaptive Acquisition