Teaching Simulators to Disagree: Amortized Human Label Entropy for Synthetic-to-Real Transfer

A simulator labels every object with complete certainty. A car that is 90% hidden behind a truck, twelve pixels tall, and blurred by motion is still labeled "car" with probability one, because the game engine knows it is there. A human looking at the same pixels would often hesitate, and several humans would split their votes. Models trained on synthetic data therefore learn a labeling rule that no real annotator follows. They become confidently wrong on exactly the real images where the evidence is thin: occluded, small, distant, or blurred objects. We argue that part of the gap between synthetic and real data is a gap in the labels, not only in the pixels. It is the gap between the simulator's certainty and human uncertainty, and it can be closed far more cheaply than the pixel gap.

The proposal is to collect several human labels on a few thousand synthetic renderings, measure how much the annotators disagree, and fit a small model that predicts this disagreement from quantities the simulator already knows for free (visible fraction, size in pixels, depth, blur, lighting). Applying that model to the full synthetic corpus gives every synthetic instance a human-like soft label at no extra annotation cost. A detector or segmenter is then trained with two heads on a shared backbone. One head predicts the simulator's true class. The other predicts the distribution of answers humans would give. The second head acts as a regularizer that should reduce overconfidence on hard real instances. If this works, two expensive things become cheap at once: multi-annotator entropy labels, which currently need 5–10 raters per item, and calibrated synthetic-to-real transfer on the hard tail.

Core Claim

Training on simulator-certain labels is a source of sim-to-real error that pixel adaptation and model-entropy methods do not touch. A soft-label auxiliary task whose targets come from human disagreement, extrapolated from simulator ground-truth factors, will improve real-domain accuracy specifically on hard instances and will improve calibration against real human vote distributions. The gain must come from the instance-specific structure of human disagreement. Generic label softening should not produce it, and neither should simply predicting the simulator factors themselves. Both are explicit controls.

Relation to Prior Work

The mechanism comes from Beyond Black & White: Leveraging Annotator Disagreement via Soft-Label Multi-Task Learning. That paper trains a shared network with a hard-label head and an auxiliary head that predicts annotator vote shares. It improves POS tagging from 85.73% to 86.76% and stemming from 73.59% to 77.09%. That work stays within a single real NLP domain and needs real multi-annotator labels for every training item. We reuse its two-head structure and its comparison of divergences, including inverse KL. The difference is that our soft targets are generated for synthetic data, and the goal is cross-domain transfer.

The claim that disagreement is structured enough to predict from instance properties is motivated by two papers. Hector Martinez Alonso et al. show that low-agreement instances correlate with feature complexity rather than annotator lapse. Lopez de Lacalle et al. find that the human vote distribution predicts model error better than static difficulty measures. HuCurl shows that human entropy is a stable difficulty signal across architectures. It uses entropy to schedule a curriculum, whereas we use it as a training target.

Two lines of sim-to-real work also use entropy or difficulty, but compute it from the model itself. TUFL shows that minimizing prediction entropy on target pixels starves hard classes and proposes a focal variant plus cross-domain image mixing. MME plays a minimax game on target prediction entropy. DisCL builds a diffusion-generated continuum from synthetic to real images and picks hard samples by model confidence. In all three, the uncertainty signal is the network's own uncalibrated output. Diagnostic Uncertainty Calibration argues that such outputs fail precisely when experts disagree. Our signal is external and human-grounded, so it complements these methods rather than replacing them. We test it as an add-on to TUFL.

Driving in the Matrix extracts certain labels from game buffers. It shows that scaling synthetic data from 10k to 200k images helps, but KITTI Hard plateaus at 0.4207 mAP versus 0.6803 on Easy. We read that plateau as the cost of certain labels on uncertain instances.

ABAW runs synthetic-data and multi-task tracks separately, with aggregated affect labels. Its synthetic-trained baseline falls from 0.50 on validation to 0.30 on real test data.

One constraint shapes the design. Improving Multi-Task Generalization via Regularizing Spurious Correlation warns that auxiliary tasks can open shortcuts. If the soft-label head mainly helps by encoding simulator factors, it could pass simulator artifacts into the shared features, which is why the factor-only ablation below is central.

Research Program

Step one: measure disagreement on synthetic images. We render about 2,000 object crops and image patches from the GTA V pipeline, stratified by visible fraction, pixel height, depth, motion blur, and illumination. Each item gets seven human labels (class plus "can't tell"). The same protocol on about 1,000 real KITTI and Cityscapes crops forms a held-out real test set for distributional evaluation.

Step two: fit the disagreement model. A small model $g$ maps simulator factors $z$ to a class distribution $q = g(z)$. We implement it as a Dirichlet-output network or gradient-boosted regressor, fitted with a smoothed likelihood. Seven votes per item give biased per-item entropy estimates, as Diagnostic Uncertainty Calibration cautions. Pooling across items with similar $z$ is exactly what reduces that bias. We then apply $g$ to every synthetic instance, using per-object targets for detection and per-pixel targets near object boundaries and on occluded regions for segmentation.

Step three: train with two heads. The objective is

$$\mathcal{L} = \mathcal{L}_{\text{task}}(f_{\text{hard}}(x), y^{\text{sim}}) + \alpha \, D\big(g(z) \,\|\, f_{\text{soft}}(x)\big)$$

where $x$ is a synthetic image and $y^{\text{sim}}$ its simulator label. $f_{\text{hard}}$ and $f_{\text{soft}}$ are the two heads on a shared backbone, and $D$ is the divergence chosen on validation (KL, cross-entropy, or inverse KL, following Fornaciari et al.). $\alpha$ weights the auxiliary loss. At test time on real images, the hard head gives predictions and the soft head gives calibrated confidence. Because $z$ is only needed at training time, real inputs never require simulator factors.

Step four: test the subjective-label regime. We port the same recipe to ABAW's synthetic-expression track. There, disagreement reflects subjectivity, which Callejas et al. show is intrinsic to spontaneous emotion, rather than occlusion alone. The factors $z$ are face size, head pose, and occlusion from off-the-shelf estimators, plus the manipulation intensity if the generator exposes it. If it does not, we regenerate a subset with our own controllable expression editor.

Baselines

The naive reference is the simulator-only model trained with hard labels. For detection, this is Faster R-CNN VGG-16 trained on Sim 200k as in Driving in the Matrix, which reports 0.6803 mAP on KITTI Easy and 0.4207 mAP on KITTI Hard. For expressions, it is the ResNet50 from ABAW, at 0.50 on validation and 0.30 on test. We rerun both on our splits, and our method adds only the soft head. Beating this reference means a significant gain on KITTI Hard over three seeds, with a smaller or zero gain on Easy. A uniform gain would point to generic regularization rather than the claimed mechanism.

The strongest existing method for dense synthetic-to-real transfer in the file is TUFL with cross-domain image mixing, from Threshold-adaptive Unsupervised Focal Loss. It reports 59.6% mIoU on GTA5→Cityscapes and 58.4% on SYNTHIA→Cityscapes with DeepLabV2. It is the strongest competitor because it already targets the same failure: hard, ambiguous target pixels being ignored. It does so with model entropy instead of human entropy. We reproduce it with the authors' settings, then add our soft head on the identical backbone, schedule, and unlabeled target data. TUFL also receives the same tuning budget we use for $\alpha$ and $D$. Beating it means exceeding the reproduced TUFL mIoU, with gains concentrated in thin and small classes (pole, sign, light, person, rider), and a lower calibration error on our real multi-annotator crops.

For the semi-supervised variant, which uses 1–3 labeled real images per class, MME is the reference. It reports 66.4% (1-shot) and 68.9% (3-shot) on DomainNet. No reported result was found on Lacuna for MME on GTA5→Cityscapes, so we run it ourselves.

Three ablations isolate the key ingredient. The first, matched label smoothing, softens every synthetic label uniformly to the same mean entropy as $g$. The second, shuffled soft labels, keeps the exact distribution of entropies but permutes them across instances, breaking the link to $z$. The third, factor heads, has the auxiliary head regress $z$ directly (occlusion, depth, blur) with no human mapping. A fourth control spends the identical human-label budget (2,000 items × 7 votes) on real images and uses them for soft-label fine-tuning.

Why not just…?

Why not just use label smoothing or temperature scaling? Both make the model less confident, but uniformly. Our hypothesis is that the benefit lies in being uncertain on the right instances: occluded and tiny ones. The matched-smoothing and shuffled-label ablations test this directly. If either matches the full method on KITTI Hard and on calibration against real human votes, the human signal adds nothing and the claim is refuted. We would report that outcome as such. Post-hoc temperature scaling cannot change the ranking of the hard head's errors, so it cannot raise Hard-tier mAP, though it may match calibration. We report both metrics separately for that reason.

Why not just predict occlusion and depth as ordinary auxiliary tasks? This is the most serious threat. Geometric auxiliary tasks are cheap, and Girshick et al. and the dense multi-task literature show that complementary heads help shared backbones. If the factor-head ablation matches our method, the human annotations are unnecessary and the honest conclusion is "use geometry heads." We expect a difference for two reasons. Human disagreement is a nonlinear, class-dependent function of the factors: a half-hidden bus is easy, a half-hidden bicycle is not. And factor regression may encourage the backbone to encode simulator-specific cues, the cross-task shortcut risk documented by MT-CRL. The comparison on real data settles which effect dominates.

Why not just collect the human labels on real images and fine-tune? With a fixed budget of 14,000 votes, labeling real images gives exactly-in-domain targets but only for 2,000 items. Labeling synthetic images lets $g$ extend the disagreement structure to hundreds of thousands of instances. This is a genuine empirical trade-off rather than a settled question, and the budget-matched real fine-tuning control measures it. If real fine-tuning wins, the useful finding is that the label gap matters but should be closed in the target domain. We also test whether $g$ fitted on synthetic crops predicts real human entropy at all, using factors estimated on the real crops. A low correlation there would explain such an outcome.

Why not just scale synthetic data further? Scaling from 10k to 200k images in Driving in the Matrix left the Easy–Hard gap at roughly 0.26 mAP. More certain labels on more hard instances do not teach appropriate uncertainty. We include a 2× synthetic-data control, sized against our method's training compute, to confirm this on our setup.

Proposed Experiments

The primary experiment is GTA V→KITTI detection and GTA5→Cityscapes segmentation, with SYNTHIA→Cityscapes as a second source. Each run reports mAP per KITTI difficulty tier, per-class mIoU, expected calibration error, Jensen–Shannon divergence to human vote distributions, and the rank correlation between predicted entropy and human entropy on the 1,000 real multi-annotator crops. All methods use three seeds and matched compute.

The secondary experiment is ABAW synthetic-to-real expression recognition. Here we also add action-unit prediction as a third head to test whether the soft head interacts with ordinary multi-task sharing. This connects the two ABAW tracks that the original challenge kept separate.

Three checks are decisive. First, the full method must beat both matched smoothing and shuffled soft labels on KITTI Hard and on the human-agreement metrics; otherwise the central claim fails. Second, it must beat the factor-head ablation, or we conclude that human labels are redundant with simulator geometry. Third, gains must be larger on hard tiers and small classes than on easy ones. As a sanity check on step two, $g$ must predict held-out synthetic human entropy well, for example with a rank correlation clearly above a size-only predictor, before any downstream result is interpreted.