Synthetic images are cheap and come perfectly labeled, but models trained on them often fail on real photographs. A common response is to treat the simulator as all-or-nothing: either pool its images with real ones, or align the two image distributions and hope. This proposal starts from a simpler observation. A simulator is often right about coarse categories and wrong about fine ones. A rendered waste stream gets "this is plastic, not metal" roughly right, because shape and context carry that distinction. It gets "soft versus rigid plastic" wrong, because that depends on surface sheen, crumpling and wear, which the renderer approximates poorly. If that is true, the right amount of trust in synthetic data depends on how deep in the label hierarchy a decision sits.
We propose to borrow the central tool of multi-fidelity modeling, the statistics of combining a cheap, biased data source with a scarce, accurate one, and apply it separately at each level of a label taxonomy. The model learns its real-world class scores as a scaled copy of the synthetic-trained scores plus a correction term learned only from real labels. Both the scaling and the size of the correction are allowed to differ by taxonomy level. The learned scales then serve two purposes. They are a measurement of where the simulator is trustworthy, and they are a safety valve: where synthetic data is useless, the scale shrinks toward zero and the model falls back to real-data-only training rather than being dragged down by it.
Core Claim
Classical multi-fidelity methods write an accurate function as $f_H(x) = \rho\, f_L(x) + \delta(x)$. Here $f_L$ is learned from abundant low-fidelity data, $\rho$ is a scale, and $\delta$ is a discrepancy term fit to the few high-fidelity points. Gaussian process classification for variable fidelity data carried this form into binary classification. It showed that $\rho \to 0$ when the cheap labels are uninformative, so the method avoids negative transfer, but it is limited to two classes and to cubic-cost Gaussian processes.
Our claim is that, for vision across the simulation-to-reality boundary, the same decomposition should be applied per node of a label tree, with level-specific trust. For a node $v$ at depth $\ell(v)$, let $z_L(x,v)$ be the score produced by a head trained only on synthetic labels. We model the real score as
$$z_H(x,v) = \rho_{\ell(v)}\, z_L(x,v) + \delta(x,v), \qquad \delta(x,v) \sim \mathcal{N}\!\left(0, \sigma^2_{\ell(v)}\right) \text{ a priori},$$
where:
- $\rho_\ell$ is the trust placed in synthetic scores at depth $\ell$;
- $\delta(x,v)$ is a correction head trained only on real labels;
- $\sigma^2_\ell$ is a per-level prior variance that controls how far real data may move the scores at that depth.
Scores at each node are turned into probabilities over that node's children. The probability of a leaf is the product of these conditional probabilities along its path, and inference sums over paths instead of committing to one hard branch at each level.
This design makes a falsifiable prediction. If simulators really are coarse-accurate and fine-inaccurate, the learned $\rho_\ell$ should fall and $\sigma^2_\ell$ should rise with depth. In that case, level-specific trust should beat a single global trust parameter, with the largest gains at small real-label budgets. If fidelity turns out to be uniform across levels, the per-level design collapses to an ordinary residual head, and the experiments will show that plainly.
Relation to Prior Work
Two papers connect label hierarchies with sim-to-real transfer, and both differ from this proposal in what they model.
TADA: Taxonomy Adaptive Domain Adaptation handles source and target label sets of different granularity. For example, synthetic "person" must split into real "pedestrian" and "rider". It does this through mixed sampling, stochastic label mapping, pseudo-label relabeling and uncertainty-weighted contrastive learning. TADA addresses mismatched label sets. We address something different: how reliable the simulator's evidence is at each level, even when the label sets match. TADA also has no explicit trust parameter that can fall to zero.
Taxonomy-Structured Domain Adaptation adds a fourth player that reconstructs a tree over domains. It proves that perfectly domain-blind features cannot also preserve an informative taxonomy. That result constrains our design. It is the reason we do not force synthetic and real features to be indistinguishable, and instead keep separate synthetic and real heads joined by an explicit correction. Note that its tree is over domains, while ours is over labels.
Three further lines of work motivate the proposal.
- VisDA 2022 Challenge: Domain Adaptation for Industrial Waste Sorting reports that top teams did better by adapting from the small real ZeroWaste-v1 set than by exploiting the 20,990-frame SynthWaste set. That is the "real data is king" paradox our safety valve is meant to resolve.
- Adversarially Masking Synthetic to Mimic Real shows that adversarial alignment can latch onto trivial cues such as whether sensor noise is present. This is a further reason to avoid relying purely on alignment.
- UrbanWaste provides a 193-class, four-level waste taxonomy and explicitly leaves open how best to exploit coarse-to-fine structure during training. It contains only real images, so it serves our hierarchy design rather than the sim-to-real question.
Research Program
The measurement study. The first deliverable is a cheap diagnostic that is useful on its own: a fidelity profile of each simulator, by taxonomy level. We train on synthetic data only and evaluate on real data. At each depth, we report accuracy conditioned on the parent being correct, and we compare it with a real-data oracle trained at the same depth. The ratio, per level, tells us where the simulator's labels carry real-world information. We will compute this profile for three pairs:
- SynthWaste → ZeroWaste, using a two-level tree built from the four ZeroWaste classes: paper (cardboard), metal, and plastic split into soft and rigid;
- GTA5 → Cityscapes, using Cityscapes' standard grouping of its 19 classes into coarse categories, plus TADA's person split;
- SYNTHIA → Cityscapes, on the 13-class protocol used by TADA.
If the profile is flat in all three, the main hypothesis is wrong, and we will report that rather than proceed.
The model. We use a single segmentation backbone (SegFormer, matching the VisDA-2022 baseline) with two heads over the same features.
- A synthetic head produces $z_L$ and is trained only on synthetic labels.
- A correction head produces $\delta$ and is trained only on real labels.
- The trust scales $\rho_\ell$ are learned jointly, as either constants per level or a small gate from features per level.
- The variances $\sigma^2_\ell$ act as level-specific weight-decay strengths on the correction head.
The variances are selected by marginal likelihood under a Laplace approximation on the last layer. This is a scalable, linear-cost stand-in for the Gaussian process machinery in the multi-fidelity classification paper, with the Gaussian process replaced by a Bayesian last layer over deep features. The backbone receives gradients from both heads, while each label source supervises only its own head. This is what stops the 20,990 synthetic frames from outweighing the 4,503 real ones in the loss, the volume-imbalance failure of naive pooling.
What gets analyzed. For each benchmark we report:
- the learned $\rho_\ell$ and $\sigma^2_\ell$ by depth;
- whether they track the measured fidelity profile;
- how performance changes as the real-label budget is swept from 1% to 100% of the labeled real set.
Baselines
Real-only training. This is the do-nothing reference and the bar implied by the VisDA-2022 paradox. It is the same SegFormer trained on the labeled real data alone, with the same hierarchical output and the same schedule. If our method cannot beat this at every label budget, the synthetic data is adding nothing.
Naive pooling. The second reference is synthetic and real labels pooled with cross-entropy, run both with and without upsampling of real images to a 1:1 ratio. The SegFormer source-only baseline in VisDA 2022 reports 45.49% mIoU on ZeroWaste-v2 (VisDA 2022). That number anchors the scale but is not a pooled result.
Synthetic pretraining followed by real fine-tuning. This is the strongest simple competitor, because fine-tuning already lets real data correct whatever synthetic training got wrong. We run it with the same backbone, the same real labels, and an equal hyperparameter search budget (the same number of trials over learning rate and schedule as our method's search over the per-level variances).
The strongest published systems. On ZeroWaste-v2, the first-place VisDA-2022 entry, SIA_Adapt with a ConvNeXt-L model soup, reports 59.66% mIoU. HRDA reports 55.50% (VisDA 2022). On GTA5 → Cityscapes coarse-to-fine, TADA reports 55.68 mIoU, and on SYNTHIA → Cityscapes it reports 59.10 mIoU (TADA). Because SIA_Adapt uses a larger backbone and ensembling, we compare in two ways. First, we report our method on its backbone, with and without a soup. Second, we rerun HRDA and TADA with our SegFormer backbone under identical data. Beating the published systems means exceeding these numbers at equal backbone and equal labeled-real data, or matching them with a fraction of the real labels.
Ablations isolating the key ingredient. These separate the per-level trust from everything else:
- a single global $\rho$ and $\sigma^2$, which tests per-level against global trust;
- a flat label space with no hierarchy, which tests whether the tree matters;
- an unconstrained correction head with no variance prior, which tests whether the prior matters;
- $\rho$ fixed to 1, which tests whether the learned trust matters.
The central claim survives only if per-level trust beats global trust.
Why not just…?
Why not just pretrain on synthetic data and fine-tune on real? This is the real threat, and we treat it as such. Fine-tuning is an implicit discrepancy model. With enough real labels it may well match ours. Our distinguishing predictions are narrower. First, at small real budgets (1–5%), fine-tuning either forgets coarse synthetic knowledge or fails to correct fine-level errors, while a prior that is tight at coarse levels and loose at fine levels avoids both. Second, our method exposes an inspectable trust parameter that falls when the simulator is useless. The label-budget sweep settles the first point directly. If fine-tuning matches us at every budget and every level, the contribution reduces to the fidelity diagnostic, and we will say so.
Why not just pool everything into a large backbone? VisDA 2022 already gives evidence against this. Its teams used ConvNeXt-L backbones pretrained on ImageNet-22K and still found that real-only adaptation beat heavy use of SynthWaste (VisDA 2022). Our pooled baseline, with and without real upsampling, reproduces this comparison under our controls.
Why not just use a flat classifier and map predictions up the tree afterwards? Evidence that the hierarchy matters is weaker than often claimed. The UrbanWaste flat baseline of 32.77 mIoU (UrbanWaste) does not by itself show that a hierarchy would do better. Our claim does not rest on hierarchical heads in general. It rests on the tree being the place where trust varies. The flat-label ablation is the test: if flat outputs with a global correction perform as well, the taxonomy is not doing the work.
Why not just apply TADA-style alignment and pseudo-labeling? These address a different gap, mismatched label sets, and they can be combined with our heads. We run TADA as a baseline and also as TADA plus our level-wise correction. Additive gains would indicate that the two methods solve complementary problems.
Proposed Experiments
The decisive checks are four.
First, the fidelity profile. Synthetic-only, conditional-on-parent accuracy relative to the real oracle should decline with depth on at least two of the three pairs.
Second, trust tracks fidelity. The learned $\rho_\ell$ should decrease with depth and correlate with the measured profile.
Third, resolving the paradox. On SynthWaste plus ZeroWaste-v1 → ZeroWaste-v2, our method should be at least as good as real-only training at full budget. Its margin should grow as the real budget shrinks, showing the synthetic set contributes positively where the challenge winners found it did not. A secondary sanity check is to replace SynthWaste with deliberately corrupted renders: $\rho$ should then fall toward zero and performance should return to real-only levels, mirroring the safe fallback reported for multi-fidelity classification.
Fourth, per-level beats global. Per-level trust should beat a single global $\rho$ and $\sigma^2$ at small real budgets, with mIoU reported per level as well as overall.
All results use three seeds, matched compute, and the same tuning budget. Metrics are mIoU overall and per taxonomy level, plus calibration error at each level, since the trust parameters are only meaningful if the per-level probabilities are calibrated.
References
- Gaussian process classification for variable fidelity data
- TACS: Taxonomy Adaptive Cross-Domain Semantic Segmentation
- Taxonomy-Structured Domain Adaptation
- VisDA 2022 Challenge: Domain Adaptation for Industrial Waste Sorting
- Adversarially Masking Synthetic to Mimic Real: Adaptive Noise Injection for Point Cloud Segmentation Adaptation
- UrbanWaste: In-the-Bin Dataset for Waste Disposal Inspection with Multi-Granularity Hierarchical Labels