Synthetic renders of animals and vehicles come with free, pixel-perfect labels for every part: head, leg, paw, wheel, mirror. Models trained on them transfer unevenly to real photographs. Whole bodies transfer reasonably well. Thin, small and deformable parts such as tails, legs and mirrors fail (Learning Part Segmentation from Synthetic Animals; Learning Part Segmentation through Unsupervised Domain Adaptation from Synthetic Vehicles). When a practitioner can afford to label a small amount of real data, the usual approach is to pick whole images to annotate. That wastes most of each annotation on parts the synthetic model already handles. This note proposes a different unit of purchase: one part-tree node in one real image, for example "outline the legs in this photo of a goat."
The selector chooses these queries under an annotation-time budget. Its objective has two properties. First, it rewards covering the real-image variety of each part node. Second, each node is weighted by how badly that node appears to transfer, and the weight is estimated without any real labels by checking where the model's part predictions contradict its own whole-object predictions. The objective is a sum of weighted coverage functions with diminishing returns (submodular), so a cost-aware greedy algorithm comes with a known worst-case approximation ratio on the selection objective. In two sentences: synthetic data already teaches most of the part hierarchy, and real labels should be spent only on the tree nodes where the model visibly contradicts itself on real images. Treating "which part, in which image" as a budgeted coverage problem over the tree makes that choice principled rather than ad hoc.
Core Claim
Under a fixed real-annotation budget, choosing (image, part-node) queries by hierarchy-aware budgeted coverage will beat choosing whole images, on part segmentation transferred from synthetic to real. This holds even when the whole-image choice uses a strong submodular active-adaptation criterion. The gain should concentrate on fine and small parts.
The claim rests on two ingredients, and each can fail independently.
The first is a label-free gap detector. A part prediction is inconsistent if no valid parent in the tree contains it. Examples are a "paw" pixel outside every predicted "leg" region, or a "mirror" pixel that the object head assigns to background. The detector's hypothesis is that the rate of such contradictions on unlabeled real images tracks the true per-node accuracy drop.
The second is the budgeted tree-coverage selector. It spreads labels across the real-image variety of the nodes that fail, instead of piling them onto near-duplicates.
The novelty is not "submodular active learning" or "hierarchical parts" alone. It is using the part-whole structure twice: once as the unit and weighting of the selection objective, and once as a self-supervised signal of where the synthetic-to-real gap lives.
Relation to Prior Work
The closest work is S3VAADA. It selects target images for labeling by greedily maximizing a submodular mix of uncertainty (prediction instability under small adversarial perturbations), diversity and representativeness. It reports a gain of about 7% accuracy on the VisDA-18 synthetic-to-real benchmark. It supplies our selection machinery. It differs from our setting in that it treats each image as one global feature vector for whole-image classification, so it cannot express that an image is worth labeling only for its legs. Its three-term objective also needs dataset-specific weight tuning. Our node weights come from a measured signal instead.
HAL3D queries human annotators coarse-to-fine down an AND/OR part tree. It reports that the tree structure alone saved 2.4 hours of labeling over a flat active learner. That finding motivates node-level queries. However, HAL3D works on pre-segmented digital meshes, has no approximation guarantee, and involves no domain shift.
CRAIG and RETRIEVE supply the facility-location coverage function and its greedy $(1-1/e)$ guarantee, following the classic analysis in Kempe and Tardos. They also constrain what we can claim. Their guarantees hold for the selection surrogate, and in deep networks they rely on last-layer proxies that void end-to-end convergence claims. We inherit that caveat and state it openly: our guarantee is about the objective, not about final segmentation accuracy. For budgets measured in annotation time rather than item count, we use the knapsack-constrained greedy results summarized in Submodular Optimization with Submodular Cover and Submodular Knapsack Constraints.
The synthetic-to-real part papers supply the unsupervised starting point we build on. Class-Balanced Fourier Data Mixing (CB-FDM) swaps image frequency statistics with real photos and balances pseudo-labels across part classes. It lifts the DAFormer baseline from 48.08 to 58.04 mean intersection-over-union (mIoU) on SynRealPart (Learning Part Segmentation from Synthetic Animals). Geometric-matching guided transfer, which warps labels from matched synthetic car views onto real cars, lifts real-car mIoU from 37.19 to 49.93 (Learning Part Segmentation through Unsupervised Domain Adaptation from Synthetic Vehicles). Both stop at a flat part list and use no real labels. We start from their adapted models and ask how to spend a small real budget on top of them.
The consistency detector borrows its logic from joint object-part decoders. TAPPS and Panoptic-PartFormer gate parts by their parent object to prevent invalid pairings. We use violations of that same constraint as a diagnostic rather than something to suppress.
Two papers pull against our design, and we take them seriously. Detect What You Can finds that under low resolution a model should switch part nodes off and rely on the whole object. PartImageNet finds that part supervision helps only when injected late in the network. Both suggest that labeling tiny parts may sometimes be wasted. Our objective accommodates this: tiny-node queries carry cost, so the greedy selector can decline them. Whether it does so is a measurable outcome, reported below.
Research Program
The candidate pool is the set of queries $q=(i,v)$ over unlabeled real images $i$ and tree nodes $v$ that the adapted model predicts as present in $i$. Each query has a cost $c(q)$, which is annotation time estimated from the predicted mask's boundary length plus a fixed per-query overhead. We will also test a second, flat per-query cost model so that conclusions do not hinge on one cost assumption.
For each node $v$, $U_v$ is the set of unlabeled real instances of that node, and $\mathrm{sim}(j,q)$ is the cosine similarity between region-pooled late-stage features of instance $j$ and query $q$. We use late-stage features following the PartImageNet observation about where part information lives. The node weight is $w_v \ge 0$, the gap estimate described below. $S_v \subseteq S$ is the set of selected queries that label node $v$ or one of its descendants; a descendant label counts at a discount $\gamma\in[0,1]$, because outlining a paw also constrains the leg. The objective is
$$f(S)=\sum_{v\in T} w_v \sum_{j\in U_v}\max_{q\in S_v}\gamma^{d(q,v)}\,\mathrm{sim}(j,q),$$
where $T$ is the part tree and $d(q,v)$ is the tree distance from the node queried by $q$ to $v$. Each inner term is a facility-location coverage function, and a nonnegative weighted sum of such functions stays monotone submodular. Greedy selection by gain-per-cost, combined with the best single affordable query, therefore carries a constant-factor guarantee under the budget constraint.
The gap weight $w_v$ is the node's hierarchy-consistency violation rate on real images minus the same rate on held-out synthetic images. The subtraction means nodes that are inherently ambiguous, even in simulation, are not over-weighted.
We will build three hierarchies. For vehicles, we group UDA-Part's 24 fine car parts under the standard 9 coarse parts, giving a three-level tree (car, coarse part, fine part). We construct this mapping ourselves and will release it. For animals, PartImageNet's own structure of 11 super-categories, 158 classes and their parts gives a real three-level tree, and SynRealPart's synthetic tigers and horses serve as the source. For scenes, Cityscapes and Pascal panoptic parts provide a scene-object-part tree, used with a TAPPS-style joint decoder.
Selection runs in rounds, for example five rounds each spending one fifth of the budget. After each round, the model is fine-tuned on the synthetic source, CB-FDM or geometric-matching pseudo-labels, and the purchased real node masks. Pixels outside the purchased masks in partially labeled images are left unsupervised rather than treated as background. This is essential for fairness: without it, partial labels would inject false background supervision.
We will measure three things: final accuracy, whether the gap detector is valid, and what the greedy selector chooses to buy at each tree depth. The last directly probes the coarse-first versus fine-first tension between HAL3D and Detect What You Can.
Baselines
The naive reference points are two. The first is the unsupervised adapted model with no real labels: CB-FDM on SynRealPart, reported at 58.04 mIoU (Learning Part Segmentation from Synthetic Animals), and geometric-matching guided transfer on UDA-Part cars, reported at 49.93 mIoU (Learning Part Segmentation through Unsupervised Domain Adaptation from Synthetic Vehicles). We re-run both in our code to get same-seed numbers. The second is random whole-image labeling at matched annotation time. Any method must clear these to justify itself. Beating the no-label reference is expected; the meaningful bar is random whole images.
The strongest existing competitor is S3VAADA's image-level submodular selection (S3VAADA). We port it to segmentation by computing its adversarial instability score on pixel predictions averaged per image. We give its three weights the same tuning budget, a fixed grid scored on a labeled validation split identical to ours, and we charge its purchased images full-annotation time under the same cost model. It is the strongest competitor because it already has submodular diversity and a synthetic-to-real track record. The only thing it lacks is part structure. No reported result found on Lacuna for S3VAADA or any active domain adaptation method on part segmentation, so we establish this number ourselves. Beating it means higher fine-part mIoU at equal annotation time, with the gain holding across three seeds and both cost models, and without losing coarse-part mIoU.
Two further baselines use the same node-level queries. The first is a region-level entropy baseline: take top-cost-normalized queries by predicted-mask entropy, with no coverage and no tree. The second is class-balanced per-part random queries, which allocates budget equally across fine parts. This second baseline tests whether "just give rare parts more labels" is the whole story.
Three ablations isolate our ingredients. Replacing $w_v$ with uniform weights removes the gap detector. Replacing $w_v$ with mean entropy tests whether consistency violations add anything over uncertainty. Setting $\gamma=0$ removes hierarchical propagation. Finally, taking the top queries by $w_v$ alone, with no coverage function, isolates the submodular diversity term.
Why not just…?
Why not just run S3VAADA on whole images? Its image-level score averages over all parts. An image whose tail is badly wrong but whose body is fine looks mildly uncertain, and labeling that image pays full cost for mostly redundant pixels. This is an argument, not evidence. The head-to-head comparison at matched annotation time is the test. If the image-level method matches us, then the node-level unit does not matter, and we would report that.
Why not just label the rarest or smallest parts, class-balanced? This is the real threat. The synthetic-to-real part papers already show that the damage sits in small parts, so a heuristic that upweights them might capture most of the gain. The class-balanced per-part random baseline and the entropy baseline exist to settle this. If either comes within seed noise of the full method, the contribution shrinks to the claim that node-level purchase beats image-level purchase. That result is still useful but smaller, and we would reframe the paper around it rather than defend the tree machinery.
Why not just prompt a foundation segmenter such as SAM on real images? Zero-shot multi-granularity masks do not respect a part tree. On Blender-HS, SAM reaches 62.7% mean normalized covering against 70.9% for an explicitly hierarchical method, and it fails to assign sub-parts uniquely to parents (View-Consistent Hierarchical 3D Segmentation Using Ultrametric Feature Fields). SAM also carries no part names. It remains useful, though: we will test SAM-assisted mask drawing as a way to lower $c(q)$, which is orthogonal to what gets selected.
Why not just add stronger appearance randomization to the synthetic data? The vehicle source images were already rendered with randomized lighting and textures, yet source-only models reached only 37.19 mIoU on real cars (Learning Part Segmentation through Unsupervised Domain Adaptation from Synthetic Vehicles). Randomization remains part of our starting model rather than a competitor to the selector.
Proposed Experiments and Decisive Checks
The first check comes before any selection and gates the whole project. On SynRealPart and UDA-Part, we compute per-node consistency-violation rates on unlabeled real images and correlate them, across nodes, with the true per-node mIoU drop from synthetic validation to real test. If the rank correlation is weak, or no better than mean entropy, the gap detector fails. We then fall back to entropy weights and present only the node-level coverage result.
The main experiment sweeps real annotation budgets. These are expressed as annotation time equivalent to labeling 1%, 2% and 5% of real training images fully, mirroring the budget range used by S3VAADA. We sweep on the PartImageNet tree with SynRealPart as the source, on the UDA-Part car tree with real PASCAL3D+ images, and on Pascal-PP. We report mIoU per tree level and Part-aware Panoptic Quality (PartPQ) on Pascal-PP, the metric TAPPS reports a 6.3-point gain on.
A useful ceiling on UDA-Part is 53.77 mIoU, the score of geometric-matching transfer given ground-truth viewpoints. It indicates how much of the gap is pure alignment rather than missing labels.
The analysis experiment records what fraction of purchased time the greedy selector spends at each depth as the budget grows. If HAL3D's coarse-first intuition holds under domain shift, early rounds should buy coarse nodes. If the Detect What You Can finding holds, the selector should decline tiny-part queries on low-resolution instances. Either outcome resolves a tension the literature currently leaves open.
Finally, we compare the greedy objective value against the best of many random cost-feasible query sets, and against exhaustive search on small subproblems. This checks that the guarantee is not vacuous in practice. We also measure whether objective value predicts downstream mIoU, which is the link that the convex-case analyses of CRAIG and RETRIEVE cannot promise for deep networks.
References
- Learning Part Segmentation from Synthetic Animals
- Learning Part Segmentation through Unsupervised Domain Adaptation from Synthetic Vehicles
- S3VAADA: Submodular Subset Selection for Virtual Adversarial Active Domain Adaptation
- HAL3D: Hierarchical Active Learning for Fine-Grained 3D Part Labeling
- Coresets for Data-efficient Training of Machine Learning Models
- RETRIEVE: Coreset Selection for Efficient and Robust Semi-Supervised Learning
- Maximizing the spread of influence through a social network
- Submodular Optimization with Submodular Cover and Submodular Knapsack Constraints
- Task-Aligned Part-Aware Panoptic Segmentation Through Joint Object-Part Representations
- Panoptic-PartFormer: Learning a Unified Model for Panoptic Part Segmentation
- Detect What You Can: Detecting and Representing Objects Using Holistic Models and Body Parts
- PartImageNet: A Large, High-Quality Dataset of Parts
- View-Consistent Hierarchical 3D Segmentation Using Ultrametric Feature Fields