Weighting Synthetic Samples by Whether They Help, Not Whether They Look Real

Models trained on simulated or rendered data usually fail when they meet real cameras, real LiDAR, or real robots. Two of the most common fixes respond to that failure by trusting simulation less across the board. One tunes a single global ratio between synthetic and real data. The other down-weights synthetic examples that a "domain classifier" (a small network that guesses whether an input is synthetic or real) flags as unrealistic. This proposal argues that realism is the wrong signal. A synthetic example should count for more when its training gradient points the same way as the gradient from a small set of labeled real examples. It should count for less, possibly near zero, when its gradient points elsewhere, however realistic it looks. The proposal builds a cheap, continuous per-sample weight from this gradient agreement and applies it directly in the training loss for synthetic-plus-few-real training.

The idea can be stated to a colleague in two sentences. When you have lots of simulated data and a little real labeled data, weight each simulated example by how well its gradient agrees with the real data's gradient, and let the weight vary smoothly between zero and one. The central test is whether this "helpfulness" weight beats a "realism" weight and a tuned global mix ratio, and whether the two kinds of weight actually disagree on which synthetic samples matter.

Core Claim

Two findings in the file motivate the claim. A Mechanistic Analysis of Sim-and-Real Co-Training in Generative Robot Policies trains robot policies on 50 real and about 3,000 simulated demonstrations. It shows that synthetic and real features which look identical in a 2-D projection can still be told apart by a linear probe with near 100% accuracy. If that holds, a realism classifier will push almost every synthetic weight toward the same extreme. Separately, OptimSyn shows for text that embedding closeness between synthetic and real data predicts downstream usefulness poorly. In that work the inner product between a synthetic sample's gradient and a real validation gradient predicts usefulness well, and superficially similar samples can have opposite effects.

Taken together, these suggest that realism-based weighting is near-degenerate in sim-to-real settings, and that gradient agreement carries the information that matters. The claim is testable in three parts. Gradient-agreement weights should correlate weakly with realism scores. They should beat both realism weights and a tuned global ratio. And the gain should come specifically from the soft, sample-specific signal rather than from any reweighting at all.

Relation to Prior Work

The proposal reuses ideas from three lines of work and changes one thing in each. Cost Weighting for Neural Machine Translation Domain Adaptation multiplies each sentence's loss by $(1 + p_d(x))$, where $p_d(x)$ is a jointly trained domain classifier's probability that the sentence is in-domain. That rule beats hard data selection by 0.8 to 1.2 BLEU. It is the realism-weighting design in its cleanest form, and it can never down-weight below the base weight; we test it directly and allow weights to fall toward zero. OptimSyn supplies the gradient-agreement signal, but uses it only as a reward for rewriting text-generation rubrics; we apply it inside the loss for visual and robotic sim-to-real training. A Mechanistic Analysis of Sim-and-Real Co-Training identifies "importance reweighting" as a secondary mechanism, but implements it only through one dataset-level ratio $w$. We replace that scalar with per-sample weights $w_i$ while keeping its domain-conditioned architecture, because that paper also constrains the design: forcing synthetic and real features to be indistinguishable destroys information the policy needs.

Other work constrains the design without supplying it. Learning from Noisy Labels via Self-Taught On-the-Fly Meta Loss Rescaling (STORM) and ATLANTIS show that meta-learned or proxy-model weights are effective but expensive. That pushes us toward a single extra gradient computation on a projected subspace. Learning with Noisy Labels Using Hyperspherical Margin Weighting and Self-Paced Learning for Matrix Factorization argue that hard keep-or-drop rules discard difficult but useful samples and destabilize optimization. On CIFAR-10 with 80% symmetric label noise, continuous margin weights reach 65.8% against 26.1% for plain cross-entropy. This motivates soft rather than thresholded weights, a choice we test rather than assume.

Incorporating Importance Weighting in Optimal Transport Based Domain Alignment weights source samples to align them with an unlabeled target distribution, which is the theoretically grounded realism view. It is demonstrated only on digit benchmarks. Automated Synthetic-to-Real Generalization (ASG) and RLS3 treat all synthetic samples uniformly or select them in a binary way, which leaves the per-sample weighting question open.

One honest difference in assumptions: the method needs a small labeled real set, whereas ASG and the optimal-transport method do not. We treat this as a deliberate choice of regime, since it matches robot co-training and practical deployment, not as a free advantage. Comparisons are kept within the same label budget.

Method

Let $g_i$ be the gradient of synthetic sample $i$'s loss, taken with respect to the last block of parameters and randomly projected to a few thousand dimensions. Projecting keeps the cost low and addresses the variance concern raised around STORM and OptimSyn. Let $\bar g_R$ be an exponential moving average of the same projected gradient on batches from the labeled real set. The weight is

$$w_i = \sigma\!\left(\tau \cdot \cos(g_i, \bar g_R) + b\right),$$

where $\sigma$ is the logistic function, $\tau$ is a temperature, and $b$ is an offset. The weights are renormalized within each batch to have mean one, so the overall synthetic learning rate is unchanged. Training minimizes $\sum_i w_i \ell_i$ over synthetic samples, plus the unweighted real loss.

The real set is split into two parts. One part computes $\bar g_R$. The other is held out for choosing hyperparameters, so the weights never see the evaluation data. For robot policies, the weighting is applied on top of the domain-label conditioning from Ko et al., which preserves the ability to tell simulation from reality.

Research Program

The work has three parts.

The first part is diagnostic. Before any method claims are made, we train a standard co-trained model and compute both the gradient-agreement score and a realism score for every synthetic sample. The realism score comes from a domain classifier on the same features, as in Chen et al. We measure their rank correlation. We also test usefulness directly: retrain on the top and bottom halves of the synthetic data ranked by each score and compare real-domain accuracy. If realism and gradient agreement rank samples nearly identically, the proposal collapses into cost weighting, and we would report that.

The second part is the method comparison described under Baselines.

The third part is analysis of what gets down-weighted. In GTA5 and VisDA we inspect classes, lighting conditions, and viewpoints. In SynLiDAR we inspect beam patterns and distance ranges. In MimicGen we inspect trajectory segments. We then check whether low-weight samples line up with the known artifact types catalogued in the file, such as fixed lighting, unrealistic point density, and simplified contact physics.

Baselines

The naive reference point is co-training with uniform weights: all synthetic data plus the same small real set, at the best global ratio found by grid search. This is exactly the global mixing ratio from A Mechanistic Analysis of Sim-and-Real Co-Training. It is the right floor because any per-sample method must beat a well-tuned scalar. We give the ratio search at least as many trials as our method's hyperparameter search. A second floor is synthetic pretraining followed by fine-tuning on the real set alone.

The strongest competitor on the specific question of which signal to weight by is realism weighting in the style of Cost Weighting for Neural Machine Translation Domain Adaptation. We run it twice: in its original $(1+p_d)$ form, and in a variant that uses the same logistic mapping and mean-one renormalization as ours. The variant isolates the signal from the functional form. It is the strongest competitor because it is cheap, has a published win over hard selection, and is the intuitive choice. Beating it means a statistically reliable gain over three or more seeds with equal tuning budgets, from a different signal applied through the same weighting function.

The strongest existing system results are benchmark-specific. On the robot suite, the comparison is the domain-conditioned plus adversarial co-training method of Ko et al., which reports 74% average success on NutAssembly, MugHang, and MugCleanup. Our weights are added on top of it, and beating it means exceeding its success rate under the same 50-real, roughly 3,000-synthetic split. On VisDA-C, ASG reports 61.1% real accuracy without real labels, per Automated Synthetic-to-Real Generalization. On SynLiDAR to SemanticKITTI, adversarial masking reports 22.8% mIoU against a 55.5% fully real oracle, per Adversarially Masking Synthetic to Mimic Real. Both are unsupervised-target numbers, so they are not directly comparable to a few-label setting. For the few-real-label regime on these vision benchmarks, no reported result was found on Lacuna, so we rerun ASG's proxy loss and adversarial masking themselves with the added real labels as baselines.

Three ablations isolate the key ingredient. The first applies a hard threshold to the same gradient score, keeping the same fraction of total weight. This tests soft against hard. The second uses random weights drawn from the same empirical distribution as ours. This rules out a generic noise-regularization effect. The third uses full-network gradients instead of projected last-block gradients on a subset. This checks whether the cheap approximation loses signal.

Why not just…?

Why not just tune the global sim-to-real mixing ratio? Ko et al. show that the ratio reshapes the shared representation, and tuned well it may capture most of the benefit. If the synthetic data are roughly uniform in quality, it will. The diagnostic study settles this. If the top-half and bottom-half retraining runs perform similarly, per-sample weighting has nothing to exploit, and the ratio baseline should match us. We report that outcome as a negative result rather than tune around it.

Why not just weight by realism with a domain classifier? This is the most serious alternative and the one the proposal is built around. The linear-probe result in Ko et al. predicts that the classifier saturates and gives nearly uniform, uninformative weights. OptimSyn shows that looking similar does not predict being useful. Both pieces of evidence come from other modalities, however, so the head-to-head comparison with the matched functional form is the deciding experiment.

Why not just pretrain on synthetic data and fine-tune on the real set? With only a few real labels, fine-tuning tends to overfit and to erase useful structure learned from simulation. ASG documents the reverse problem, where training on synthetic data erases real-image priors. Still, this baseline is cheap and sometimes hard to beat, so we include it at every label budget. If it wins at the larger budgets, we report the label budget below which weighted co-training helps.

Why not just discard the unhelpful synthetic samples? The hard-threshold ablation tests this directly. Evidence from noisy-label learning favors soft weights, but in sim-to-real a simulator may produce samples that are simply wrong, and for those a hard cut may be equally good.

Proposed Experiments

The benchmarks are VisDA-C (about 152,000 CAD renders to about 55,000 COCO images, using 10 and 50 labeled real images per class), GTA5 to Cityscapes (about 25,000 frames, using 100 and 500 labeled Cityscapes images, scored by mIoU on the Cityscapes validation set), SynLiDAR to SemanticKITTI and nuScenes (a small labeled scan subset, scored by mIoU), and the Robosuite co-training setup with 50 real and about 3,000 MimicGen trajectories, scored by task success rate.

The decisive checks, in order, are these. First, the rank correlation between gradient-agreement and realism scores should be low, and the top-half retraining should beat the bottom-half retraining under gradient ranking by more than under realism ranking. Second, gradient-agreement weighting should beat the matched-form realism weighting and the tuned global ratio on at least three of the four benchmarks, with consistent signs across seeds. Third, soft weights should beat both the hard-threshold and random-weight ablations. Fourth, wall-clock overhead should stay under about 30% over uniform co-training. If the first check fails, the rest of the proposal reduces to Chen et al.'s cost weighting applied to new domains, and we would report it as such.