Factories increasingly want vision models that recognize parts and assemblies before a production line exists, which means training only on renders of the computer-aided design (CAD) files. On isolated parts this works reasonably well. On assembled products it falls apart. The SIP-17 benchmark shows a randomized-render classifier reaching 83.2% on single parts but only about 50% on wheel assemblies and 40% on engine assemblies. Its authors trace the failure to two causes. Models classify by surface color instead of shape, which they call the "albedo trap". And they cannot connect a tiny critical sub-part, such as a screw, to the structure it belongs to (Towards Sim-to-Real Industrial Parts Classification with Synthetic Dataset). We propose to generate synthetic training data whose variety is organized by the assembly's own technical vocabulary. That vocabulary is the bill of materials and the "part-of" and "fastened-to" relations already present in the CAD hierarchy. We render partial assemblies, exploded views, missing-part variants, and occlusions that follow those relations. Material appearance is assigned per part role, so that color carries no class information.
The bet is that the assembly collapse is a coverage problem along the compositional axis, not a photorealism or texture problem. If so, spending a fixed rendering budget on structured compositional variety should beat spending the same budget on more nuisance variety: backgrounds, noise, and random materials. This tests in a concrete industrial setting a claim the dataset-diversity literature mostly states qualitatively: which axis of diversity you buy matters more than how much you buy.
Core Claim
The claim has three parts, each falsifiable on SIP-17's real test photos.
First, the albedo trap is a correlation between material and class in the training renders. It can be measured with a simple probe and removed by assigning materials per part role rather than per part identity.
Second, removing the albedo trap alone will not recover assembly accuracy. The remaining gap comes from compositional coverage: the model never sees an assembly with its sub-parts in varied states of presence, occlusion, and scale.
Third, using the CAD part-of graph to drive that compositional variety recovers most of the assembly gap. Equal-budget random part placement does not. The graph also supplies auxiliary sub-part labels, which add to the benefit.
The domain vocabulary enters as structure, not as text prompts. Technical part names are poorly known to web-trained models, and we treat that mismatch as a reason to take the vocabulary from the engineering source rather than ask a language model to invent it.
Positioning Against Prior Work
Three papers motivate the design. Structured Domain Randomization procedurally places objects along road splines so scenes stay contextually coherent. It beat unstructured randomization on KITTI hard car detection, 52.2 versus 24.0 AP, and also beat photorealistic game data (Structured Domain Randomization: Bridging the Reality Gap by Context-Aware Synthetic Data). We keep its central lesson, that context-respecting randomization beats free randomization. We replace its hand-coded geometric rules with relations read from the product's assembly graph, which is the natural "context" for a mechanical part.
AttrSyn uses a large language model to enumerate attribute values for text-to-image prompts. It improves zero-shot fine-grained classification to 56.51% on CUB-200-Painting (Attributed Synthetic Data Generation for Zero-shot Domain-specific Image Classification). It needs a human to filter incoherent attribute combinations, and it plateaus around 15k–20k images. We avoid both limits by taking attributes and their valid combinations from the CAD hierarchy, which is coherent by construction, and by rendering with a physics-based renderer rather than relying on a diffusion model's limited knowledge of technical objects.
Quality-Diversity Generative Sampling fills a grid of attribute combinations by searching a frozen generator's latent space (Quality-Diversity Generative Sampling for Learning with Synthetic Data). We borrow its idea of an explicit coverage archive. Here the archive is defined over assembly states, which are discrete and enumerable, so it can be covered directly. We do not assume a generator that already "knows" the concepts.
One paper constrains the design. The spacecraft distillation work shows that a small student trained on vision-language model pseudo-labels can beat its teacher, by 25 AP75 points on SPARK-2024 (Annotation Free Spacecraft Detection and Segmentation using Vision Language Models). That pipeline needs unlabeled real images and a teacher that understands the class name. Both are absent before commissioning, and both are doubtful for names like "engine assembly UC6 variant". We therefore treat it as a competing route to test, not a component.
BioVLN shows the same pair of failures in another setting. Zero-shot VLM navigation fell to 19.4% success because of unfamiliar apparatus names and flat synthetic textures (BioVLN: A Simulation Platform for Visual Language Navigation in Biomedical Laboratories). This supports our separation of the appearance problem from the semantic and compositional problem.
Research Program
We build a Blender-based renderer over the SIP-17 CAD models. A known risk is that the public release may not include assembly hierarchies for use cases 5 and 6. If so, we reconstruct the part-of and fastened-to graph from the CAD files or by manual annotation, which takes hours for 17 parts.
From the graph we sample assembly states. Each state specifies which sub-parts are present, which are partially exploded along their fastening axis, and which are occluded by neighbors. The camera is placed so that the smallest functional sub-part spans a controlled range of pixel sizes.
Materials are drawn from a library indexed by part role (fastener, housing, rotating element). Within that library, each class receives the same distribution over albedo, roughness, and metalness, so that mean color predicts class at chance.
Every render carries the class label plus a multi-label vector of visible sub-parts. The classifier gets an auxiliary head that predicts this vector. The point is to force attention onto small components.
Two diagnostics come with the data. The first is the shortcut probe: a logistic regression on each image's mean and histogram of color, trained on renders and tested on real photos. It tells us how much class information the appearance channel carries. The second is a coverage score: the fraction of reachable assembly states, binned by sub-part visibility and pixel scale, that appear in a training set. This gives us a quantitative, task-aligned diversity index, which the diversity literature generally lacks.
Baselines
The naive reference point is a ConvNeXt trained on SIP-17's clean renders, Syn_O, evaluated on the real photos. SIP-17 reports it far below the randomized variant but gives no per-use-case figure we can quote.
The strongest existing method is the SIP-17 recipe itself: ConvNeXt trained on Syn_R, which adds background, blur, noise, and tint randomization. The paper reports 83.2% accuracy on isolated parts, roughly 50% on wheel assemblies, and 40% on engine assemblies. We rerun it from the released data so that every comparison uses the same 566 real test photos, the same architecture, the same image count, and the same hyperparameter search budget, set at 20 trials selected on a held-out synthetic validation split, never on real images.
Beating this baseline means a gain on use cases 5 and 6 large enough to exceed the binomial uncertainty of the small real test set, with isolated-part accuracy no worse. We target at least a 15-point gain on assemblies.
We run the same comparison with a frozen DINO feature extractor plus a linear probe, because SIP-17 benchmarked DINO and AttrSyn found that vision transformers transfer synthetic global structure better.
Two ablations isolate the key ingredient. The first adds role-indexed material randomization to Syn_R with no compositional sampling. It tests whether fixing the albedo trap alone closes the gap. The second matches our compositional variety in quantity, but samples part placements uniformly rather than from the assembly graph, in the spirit of naive domain randomization. It tests whether the graph structure, and not just more cluttered multi-part images, is responsible. A third ablation removes the auxiliary sub-part head.
Why not just…?
Why not just randomize albedo uniformly, without any vocabulary? This is the most serious threat, and the design isolates it with the first ablation. If uniform or role-indexed material randomization alone recovers assembly accuracy, then the albedo trap was the whole story and the compositional claim is wrong. We would report that. SIP-17's diagnosis names two separate causes, small sub-part footprint and albedo. Material randomization cannot plausibly create views in which a screw is visible at varied scales and occlusion states, so we expect it to fix isolated-part confusions but not assemblies.
Why not just render more photorealistically? Structured Domain Randomization found procedurally varied synthetic data transferred better than photorealistic GTA V captures. SIP-17 found clean, realistic renders worse than noisy randomized ones. We include a high-fidelity path-traced render arm at equal image count to confirm this in the industrial setting.
Why not just fine-tune on a handful of real photos? Before commissioning, no real photos exist, which is the setting we target. Where some exist, this is a fair competitor. We hold out a stratified fifth of the SIP-17 real photos and compare few-shot fine-tuning from Syn_R against few-shot fine-tuning from our data. The question is whether our gains persist or vanish once real labels arrive. If they vanish at very small shot counts, the method's value is limited to the zero-real-data regime, and we will say so.
Why not just pseudo-label real images with a vision-language model and distill, as in the spacecraft work? That pipeline worked with a single coarse concept the teacher already knew. We run GroundedSAM-2 and CLIP zero-shot on SIP-17 using the technical class names and part descriptions. No reported result found on Lacuna for zero-shot VLMs on SIP-17. If the teacher's accuracy on assemblies is near chance, distillation has nothing to amplify. If it is surprisingly good, we will test distillation from it as an additional baseline.
Proposed Experiments
The decisive experiment trains every arm on 33,000 renders, matched to SIP-17's training size, and reports per-use-case accuracy on the real photos with Wilson confidence intervals. We do this for ConvNeXt, ViT, and frozen-DINO probes, over three seeds each.
Our predicted pattern is as follows. Role-indexed materials alone should drive the shortcut-probe accuracy to near chance and lift isolated parts, while leaving use cases 5 and 6 near 40–50%. Graph-driven compositional sampling should lift those two use cases substantially. Unstructured multi-part placement at equal budget should lift them much less.
A scaling sweep from 5k to 60k renders, for both Syn_R and our generator, tests whether compositional coverage avoids the early plateau AttrSyn saw. We plot real accuracy against the coverage score to see whether coverage, rather than image count, predicts transfer.
A failure analysis buckets real assembly photos by the pixel area of their smallest distinguishing sub-part. If the method works for the stated reason, the gain should be largest in the smallest-footprint bucket.
Because 566 test images is a thin basis for strong claims, we photograph a second real test set of the same assemblies under different factory lighting. We pre-register the ablation predictions before evaluating on it.
References
- Towards Sim-to-Real Industrial Parts Classification with Synthetic Dataset
- Structured Domain Randomization: Bridging the Reality Gap by Context-Aware Synthetic Data
- Attributed Synthetic Data Generation for Zero-shot Domain-specific Image Classification
- Quality-Diversity Generative Sampling for Learning with Synthetic Data
- Annotation Free Spacecraft Detection and Segmentation using Vision Language Models
- BioVLN: A Simulation Platform for Visual Language Navigation in Biomedical Laboratories