Specification Bottlenecks for Sim-to-Real Industrial Part Recognition

Factories increasingly want to train inspection models only on images rendered from their own computer-aided design (CAD) files, the 3D engineering models parts are designed in. That would let them inspect a part before a single physical copy exists. These models recognize isolated parts reasonably well in the real world. They fail in two specific ways. First, they confuse parts that share a surface finish, such as black hooks and black plugs, because color was an easy cue in the renders. Second, their accuracy collapses when small parts sit inside larger assemblies (Towards Sim-to-Real Industrial Parts Classification with Synthetic Dataset). The same CAD files that produce the renders also contain a written specification of every part: its proportions, hole and thread counts, material, and which assembly it belongs to. Current sim-to-real methods throw this specification away and keep only the pixels.

We propose to force the recognizer to pass through that specification. The network does not predict a class label directly. It predicts a short vector of physical attributes read from the CAD file, chosen because they separate parts that are easily confused and because they do not depend on how the image was rendered. The class is then found by matching the predicted vector against the specification sheet. When a few real images are available, the network's features are aligned between synthetic and real data separately for each attribute, rather than for each class label. The claim is that a specification bottleneck removes rendering shortcuts that domain randomization leaves in place. It should also let the system recognize a part that is new in CAD with no new training, simply by adding a row to the specification sheet.

Core Claim

A sim-trained classifier can use any cue that predicts the label in the renders, including albedo (surface color under neutral lighting) and rendering artifacts. A classifier whose only route to the label is through attributes such as "aspect ratio," "number of through-holes," "thread present," or "member of the wheel assembly" can use a shortcut only where the attribute itself is predictable from that shortcut. By choosing attributes that are deliberately uncorrelated with surface finish among parts that share a finish, we remove the albedo shortcut by construction. Randomizing albedo also attacks this shortcut, but only in expectation and only for the cues one thought to randomize.

The bottleneck also changes what domain alignment conditions on. DIRL showed that aligning synthetic and real features globally maps different classes on top of each other, a failure called negative transfer, and fixed it with discriminators conditioned on one-hot class labels. We keep that insight and condition alignment on attribute values instead. Attributes are shared across classes, so each attribute discriminator sees many more real examples than a per-class discriminator. This should matter most in the regime of 5–10 real images per class that DIRL targets.

We commit to a falsifiable expectation. On the real test images of the SIP-17 benchmark, the bottleneck should reduce confusions between same-finish part pairs more than albedo-randomized rendering does. It should also raise accuracy on the assembly use cases, where the source-only accuracy is reported as roughly 50% for wheels and 40% for engines.

Relation to Prior Work

Three papers motivate the gap. Zhu et al. built SIP-17 and documented both the albedo trap and the assembly collapse. They treat the problem as purely visual and suggest heavier material randomization; we test that suggestion directly as our main competitor. Nosrati et al. show that engineering metadata helps in practice. They feed a surgical tray's parts list into the detector's final filtering step, which discards duplicate and overlapping boxes. They also add a loss on box area to separate look-alike screws by size. However, their detector was trained on over 130,000 real frames, and the metadata enters only after the network has already made its predictions. Our proposal moves the metadata into training and removes the need for real data. The error-notebook CAD retrieval work shows that text descriptions of CAD geometry help a vision-language model pick out subparts. It works only on clean renders, with frozen GPT-4o and multi-second latency. We borrow its idea of turning heavy CAD into compact descriptors, but we use structured attributes rather than free text, and we test them on real images.

The method draws on two sources. DIRL supplies the conditional-alignment machinery, which we re-key from class labels to attributes. The Fusion 360 Gallery (Fusion 360 Gallery) supplies the representation of CAD models as boundary-representation face graphs. We use these graphs to compute attributes automatically: hole and boss counts from cylindrical faces, proportions from bounding boxes, and material from part properties.

Three lines of work constrain the design. PixelDA, which translates synthetic images into the real style with a generative adversarial network, states that it handles low-level appearance and not structural change. Adversarial spatial masking, which learns where to drop points from synthetic LiDAR scans, shows that domain discriminators latch onto rendering artifacts. Both warn that unconstrained adversarial alignment is fragile, and this is why we condition the discriminators. The chart-grounding results in ChartLens warn of the opposite risk: language-heavy models can ignore fine visual detail. Our bottleneck avoids this because the metadata is never an input at test time. It is only a supervised target and a lookup table, so the model cannot shortcut through it.

Research Program

The first step is attribute selection. For each part in a catalog, we extract a candidate pool of attributes from CAD: relative proportions, counts of holes, slots and teeth, thread presence, rotational symmetry order, material class, color, and assembly membership with parent and sibling identities. We then choose a subset greedily. Each chosen attribute must separate many confusable pairs; confusable pairs are those a source-only model mixes up on held-out renders. Its correlation with rendered color among parts that share a finish must be near zero.

We use relative attributes rather than absolute size by default, because SIP-17 images have no known camera distance. The area loss of Nosrati et al. shows that absolute scale is powerful when a fixed station makes it observable, so we add it as a variant for that setting.

The second step is the recognizer. A ConvNeXt backbone, matching the strongest architecture reported on SIP-17, feeds one small prediction head per attribute. A class is chosen by maximum likelihood of the predicted attributes under each part's specification row. Optionally, the class can be restricted to parts listed in the assembly manifest, which is the in-training version of Nosrati's inference-time filter.

The third step is the alignment variant for the few-label regime. It keeps DIRL's global discriminator and triplet loss but replaces class-wise discriminators with attribute-wise ones.

The fourth step is analysis. We measure which attributes transfer, comparing real-versus-render error per attribute. We also test whether assembly-membership attributes, which tell the model which subparts should co-occur, explain the gains on assemblies.

Baselines

The naive reference is ConvNeXt trained on the randomized SIP-17 renders and tested on the 566 real images. Zhu et al. report 83.2% accuracy on isolated parts for this setup, roughly 50% on wheel assemblies and 40% on engine assemblies. We re-run it with our seeds so that every comparison uses identical splits.

The strongest competitor on the albedo question is the remedy Zhu et al. themselves propose: domain randomization extended to randomize albedo, material and texture per part. No reported result found on Lacuna for albedo-randomized SIP-17. We render it from the same CAD files at the same image count and with the same backbone. We give it the same tuning budget as our method, measured in hyperparameter trials and GPU hours. Beating it means a statistically significant reduction in same-finish pair confusions together with no loss on isolated parts.

For the parts-list question, the competitor is the source-only model with a manifest restriction applied only at inference, following Nosrati et al.. No reported result found on Lacuna for this on SIP-17.

For the few-label regime, the competitor is DIRL with the same 5–10 real images per class. DIRL reports going from 26.8% to 91.0% recognition accuracy on the Toyota robot benchmark. We re-implement it on SIP-17 and, if that data can be obtained, on the robot benchmark. Our method must beat it at equal label counts.

We include PixelDA translation of the renders as the generative alternative. We include zero-shot CLIP-style matching of real images against text renderings of the specification rows as the foundation-model alternative.

The key ablation keeps the bottleneck architecture but trains it on randomly permuted attributes of matched cardinality. This separates the benefit of real physical specifications from the benefit of having a multi-head, structured output. A second ablation adds color to the attribute set, to show that reopening the shortcut reopens the trap.

Why not just…?

Why not just randomize albedo and materials more heavily? This is the real threat, and it may close much of the same-finish gap on isolated parts. Randomization only removes the cues one chose to randomize, whereas the bottleneck removes every cue not tied to a specified attribute. Randomization also does nothing for onboarding new parts or for subparts inside assemblies, where Zhu et al. saw the collapse even under randomized rendering. If albedo-randomized rendering matches our method on same-finish confusions, we will narrow the claim to assemblies and onboarding new parts from CAD alone and report that honestly.

Why not just filter predictions with the manifest at inference? A filter can delete classes that should not be present, but it cannot recover a part the network never scored highly. On SIP-17 each use case already involves few candidate parts, so filtering may help little exactly where the confusions occur, between parts that legitimately share an assembly. Comparing this baseline with our in-training manifest variant isolates the difference directly.

Why not just prompt a large vision-language model with the specification? GPT-4o reached only 41.7% zero-shot on clean CAD part retrieval, and 65.1% with elaborate self-correction (error-notebook retrieval). Latency was 6.5–8 seconds per query, against under one second for an on-device detector in Nosrati et al.. Our CLIP-style zero-shot baseline measures the open-weight version of this approach on real images.

Why not just use DIRL with class labels? DIRL is strong when every class has some real labels. However, it cannot name a class that has no real examples, and a per-class discriminator sees only its own class's few images. The attribute-conditioned variant and the leave-one-part-out test below compare the two directly.

Proposed Experiments

The main experiment trains on the SIP-17 renders, both the clean and randomized versions, and reports real-image accuracy overall, per use case, and on a pre-registered list of same-finish pairs taken from the confusion analysis of Zhu et al. With only 566 real images, we report bootstrap confidence intervals and McNemar tests rather than single numbers.

The onboarding experiment holds out one part at a time. It trains attribute heads on the remaining parts and classifies the held-out part's real images purely by specification matching. No class-label baseline can do this at all, so the comparison is against the CLIP-style matcher. We repeat the attribute-extraction pipeline on Fusion 360 assemblies, using renders only, to check that the extraction scales beyond 17 parts.

The few-label experiment sweeps 0, 1, 5 and 10 real images per class and compares attribute-conditioned alignment with DIRL.

The decisive checks are three. The first is whether the bottleneck beats albedo-randomized rendering on same-finish pairs. The second is whether permuted attributes erase the gain. The third is whether accuracy on the wheel and engine assemblies moves meaningfully above the reported roughly 50% and 40%. If the first check fails but the third succeeds, the contribution is that engineering metadata fixes assembly collapse rather than color shortcuts. That result is still worth knowing.