Descending the Taxonomy: Coarse-First Self-Training for Synthetic-to-Real Transfer

Models trained on rendered or generated images often fail on real photos. The failure is uneven, though. A classifier trained on game-engine frames may still tell "vehicle" from "vegetation" in a real street scene while confusing "truck" with "bus." We propose to use this unevenness during adaptation. The model teaches itself on unlabeled real images by assigning its own guessed labels, and it assigns each real example only as deep a label in the class hierarchy as it can trust. A real image the model is sure is a "vehicle" but unsure between truck and bus is trained toward "vehicle": the loss is spread over that whole branch of the label tree instead of the example being thrown away or forced to a possibly wrong leaf. Across training, the allowed depth moves from broad categories to fine ones.

In two sentences: guessed labels on real data are much more reliable near the root of a taxonomy than at its leaves. So self-training across the synthetic-to-real gap should start at the root and move downward, keeping partially confident examples as coarse labels instead of discarding them. The work first tests the premise directly, by measuring whether the synthetic-to-real accuracy drop grows with taxonomic depth. It then tests whether a depth-ordered, partial-label version of self-training beats flat self-training, frequency augmentation, and test-time entropy minimization on the same backbone and data.

Core Claim

The claim has two parts, and each can fail on its own. The empirical part is that the synthetic-to-real gap widens with depth in the label tree. This is plausible but untested. PASTA locates much of the gap in high-frequency image content that simulators under-vary, and fine-grained distinctions plausibly rely on that content more than coarse ones. Curriculum Domain Adaptation found that coarse, image-level properties such as label proportions can be inferred on real images more reliably than pixel labels. Neither paper measures the gap as a function of depth.

The methodological part is that if the gap does widen with depth, the right unit of self-training is the deepest trustworthy ancestor, not the leaf. The key ingredient is the partial pseudo-label. Ordering by depth is secondary. Flat self-paced self-training, in the spirit of Self-Paced Learning, already starts with easy examples. It can only accept or reject a leaf label, though. The hardest real examples are exactly those with uncertain leaves, so flat methods discard most of the fine-grained signal early and later admit it as confident errors. Tent is known to reinforce such errors, because it pushes predictions toward whatever the model is already confident about, including when it is wrong.

Concretely, let $p_\theta(c \mid x)$ be the model's leaf-class probabilities for a real input $x$ (an image or a pixel). For each tree node $v$, let its marginal be the sum of $p_\theta$ over the leaves beneath $v$. Let $a(x)$ be the deepest node whose marginal exceeds a depth-specific threshold $\tau_{d}$, among the depths currently admitted by the schedule. The target-side loss is

$$\mathcal{L}_{\text{tgt}}(x) = -\log \sum_{c \in \text{leaves}(a(x))} p_\theta(c \mid x),$$

which pushes probability mass into the trusted branch without choosing among its children. Source images keep ordinary leaf cross-entropy. The schedule admits depth $d+1$ once the share of target examples that are confident at depth $d$ stops growing.

Relation to Prior Work

Three papers already join curricula with hierarchies, and all of them assume training and test data come from the same distribution. Hierarchical Class-Based Curriculum Loss bounds each child's loss by its parent's and introduces fine classes gradually. HiCu warm-starts fine ICD-code classifiers from coarse ones. Coarse-to-Fine Curriculum Learning builds a hierarchy from confusion patterns. These papers supply the idea that depth can order training. We move that ordering to the unlabeled target side of a domain shift, where depth controls how reliable the model's own labels are, not how hard the task is to learn.

Two papers join curricula with sim-to-real transfer, and both treat classes as flat. Curriculum Domain Adaptation uses inferred label proportions and superpixels as easy target tasks. Diffusion Curriculum interpolates from synthetic to real images with guided diffusion. Our curriculum runs over the output space rather than over inputs or auxiliary tasks, so it can be combined with either.

PASTA and Towards Sim-to-Real Industrial Parts Classification supply the motivating evidence that fine detail is where transfer breaks. The industrial paper reports accuracy falling from 80–90% on isolated parts to roughly 40–50% on assemblies despite domain randomization. We use PASTA as the default augmentation under every method, so our gains are measured on top of it. Tent constrains the design: its own authors note that entropy minimization can reinforce overconfident mistakes, and our partial labels are meant to avoid exactly that.

Research Program

The first stage is a diagnostic, and it can end the project. We train source-only models (DeepLabv3+ with PASTA for segmentation, a ConvNeXt with PASTA for classification) and measure accuracy at every tree level on both held-out synthetic data and real data. We also measure how precise pseudo-labels are at each level, as a function of confidence.

We use hierarchies that exist independently of this project so the tree is not tuned to the result. For segmentation, these are the official Cityscapes category groups (flat, construction, object, nature, sky, human, vehicle) above the evaluation classes. For fine-grained classification, these are the manufacturer, family and variant levels of FGVC-Aircraft. For industrial transfer, these are the use-case and assembly groupings of SIP-17. If the real-minus-synthetic accuracy gap and pseudo-label error do not grow with depth on at least two of the three settings, the premise is false and we stop there. That negative result would still be worth reporting.

If the premise holds, the second stage builds the depth-gated self-training described above and compares it with the baselines below. The third stage is analysis. We audit pseudo-label precision per depth against held-out real labels over the course of training. We measure how many target examples each method uses. We check whether the method's mistakes are confined to siblings, measured by the hierarchical distance of errors as defined in the HCL paper.

Baselines

The naive reference point is source-only training with PASTA augmentation and flat cross-entropy. It is a strong naive point. On GTAV→Cityscapes, PASTA reports DeepLabv3+ improving from 28.95 to 44.12 mIoU, and it is the strongest single-number baseline in our file for that benchmark. We rerun it with our own backbone and schedule instead of copying the number. Any method must beat our rerun, and we expect at least parity with the published 44.12.

The strongest competitor on the actual question is flat self-paced self-training. It uses the same PASTA-augmented source model and the same unlabeled real images. It applies per-class confidence thresholds on leaf predictions that loosen over training, and it discards everything below threshold. It gets the same number of target passes and the same threshold-tuning budget as our method, a small grid tuned on a held-out real validation split used identically for all methods. This matters because the VisDA challenge winner reached 92.8% on VisDA-C with pseudo-labeling, showing flat self-training is already powerful. Our method beats it only if it improves leaf-level real accuracy or mIoU beyond run-to-run variance across three seeds, and not merely hierarchical-distance metrics.

We also run three further baselines. The first is Tent on the same source model; its authors report GTA5→Cityscapes IoU rising from 28.8% to 35.8%. The second is HCL trained on synthetic data only, which tests whether a hierarchy-aware loss without target data is enough. The third is, on SYNTHIA→Cityscapes, Curriculum Domain Adaptation, which reports 29.0% mIoU against a 22.0% unadapted baseline.

For SIP-17, the reference is the reported 83.2% ConvNeXt accuracy on isolated parts and roughly 40–50% on assemblies from the industrial parts paper. For FGVC-Aircraft trained only on generated images and adapted on unlabeled real images, there is no reported result on Lacuna for that exact setting. BOB reports 57.4% but with five labeled real images per class, so it is not a direct comparison.

Three ablations isolate the ingredients. The first removes partial labels: it keeps the depth schedule but discards any example not confident at the leaf. This separates the partial-label effect from ordering. The second removes the schedule: all depths are open from the start with the same deepest-trusted-ancestor rule. The third uses a shuffled tree with the same shape but leaves randomly reassigned to parents. This tests whether semantic structure matters, or whether any set-valued labeling helps.

Why not just…?

Why not just use flat confidence-thresholded self-training with a slow threshold schedule? This is the real threat. If uncertain real examples are uncertain everywhere in the tree rather than mainly at the leaves, a flat method loses nothing by discarding them, and our approach reduces to it. The no-partial-labels ablation and the per-depth precision audit settle this directly. We will report the result even if the flat method matches ours. A match would mean the diagnostic's depth gradient exists but is not exploitable.

Why not just rely on stronger augmentation such as PASTA or domain randomization? Augmentation changes the inputs the model sees but never uses real images. The industrial parts results show randomized renders still collapse on assemblies. Because PASTA sits underneath every method here, the question becomes whether target-side self-training adds anything on top of it. The comparison with source-only PASTA answers that.

Why not just adapt at test time with entropy minimization? Tent needs no labels and little compute. It pushes all confident predictions toward certainty, including wrong leaves, which its authors acknowledge. We run it on identical source models. The comparison is most telling on FGVC-Aircraft variants and SIP-17 assemblies, where confident sibling errors should be common.

Why not just train with a hierarchical loss on synthetic data? HCL gives consistent predictions on the domain it was trained on. Nothing in its evaluation touches a domain shift, and consistency on synthetic data does not repair features that are missing on real images. Synthetic-only HCL is a baseline. If it closes most of the gap, the target-side mechanism is unnecessary.

Decisive Checks

The project is judged on three results. First, the phase-one depth diagnostic must show a gap that widens with depth in at least two settings. Second, leaf-level real accuracy or mIoU must exceed flat self-paced self-training beyond seed variance on GTAV→Cityscapes and at least one classification setting. Third, the partial-label ablation must account for most of that gain, and the shuffled tree must lose it. If the shuffled tree performs as well as the true one, the contribution is set-valued self-training, not the hierarchy, and we will say so.