Symmetry-Aware View Profiles for Rejecting Unknown Objects in CAD-Based Pose Pipelines

Robots that pick parts from bins increasingly rely on "CAD-based" pipelines. These systems take only a 3D model of each part, render it from many viewpoints, and find the part in a camera image by comparing image crops against the rendered views. Such pipelines assume every object in the bin is in the catalogue. Real bins break that assumption. They contain unseen bolts, near-duplicate brackets, and defective parts, and these unknown objects get matched to the most similar known part. This proposal asks how such a pipeline should decide that a crop belongs to none of its catalogued objects. It argues that the answer depends on geometric symmetry.

Today, the score that decides whether a crop matches an object is the best similarity, or the average of the few best similarities, between the crop and that object's rendered views. Symmetry distorts this score. A cylinder looks identical from many viewpoints, so its top few rendered views all tie at the peak. An asymmetric bracket's top few include neighbouring, less similar views. The same rejection threshold therefore means different things for different objects, and lookalike unknowns that happen to be symmetric are favoured. We propose to replace the single-number score with a richer "prototype." For each rendered view of a known object, we record how similar that view is to every other view of the same object. This similarity pattern is shaped by the object's symmetries. A genuine view of the object should reproduce one of these patterns, while a lookalike that resembles the object from one side should not.

Core Claim

The claim has two parts. The first is diagnostic. The mean-of-top-5 cosine score used by CNOS, which matches crops proposed by the Segment Anything Model against DINOv2 descriptors of 42 rendered views, is systematically inflated for objects with high symmetry order. As a result, a single global threshold over-accepts unknown objects that resemble symmetric known parts.

The second part is constructive. We define a training-free rejection score that compares patterns of similarity across views rather than individual similarity values. Let $T_1,\dots,T_N$ be the rendered views (templates) of object $k$, and let $\phi$ be an orientation-sensitive descriptor. For a query crop $x$, its view profile is the vector $s(x) = (\cos(\phi(x),\phi(T_j)))_{j=1}^N$. Offline, we compute the object's own self-similarity rows $K_i = (\cos(\phi(T_i),\phi(T_j)))_j$. We then tie each row across symmetry-equivalent views, averaging $K_i$ over all $T_{g\cdot i}$ with $g$ in the object's symmetry group $G$. This averaging removes rendering noise without erasing the repeated peaks that symmetry creates. The novelty score is

$$ r_k(x) \;=\; \min_i \; d\big(z(s(x)),\, z(\bar K_i)\big), $$

where $\bar K_i$ is the symmetry-tied row, $z(\cdot)$ standardizes a profile to zero mean and unit variance, and $d$ is Euclidean distance. A crop is rejected as unknown if $\min_k r_k(x)$ exceeds a threshold. The standardization is deliberate. Crops from real images are systematically less similar to renders than renders are to each other, but the shape of the profile should survive this gap better than its level does. Symmetry enters as structure: a cylinder's prototype rows contain a ring of equal peaks, a part with two-fold symmetry has paired peaks, and an asymmetric part has a single peak. An unknown object that matches one view well but not its symmetric partners or neighbours produces a profile with the wrong shape.

How This Relates to Prior Work

Prototype-based open-set recognition supplies the general idea of rejecting by distance to bounded class regions rather than by softmax confidence. Towards Open World Recognition rejects queries far from every class mean. Adaptive Decision Boundary learns a spherical radius per class on frozen features, and Multi-Relation Margin loss learns radii and margins around prototypes. All three treat each class as one roughly round blob in feature space. A 3D object observed from all viewpoints is not a blob. Its views trace out a curved set whose shape is fixed by the object's symmetries, and averaging those views collapses toward a meaningless point. The pose-regression literature documents the same failure as "mean collapse" for symmetric objects (3D Pose Regression Using CNNs, On Object Symmetries and 6D Pose Estimation). This motivates using one prototype per view and making those prototypes relational: each is a pattern over all views rather than a single feature vector.

The CAD-based pipelines supply the setting and constrain the descriptor choice. CNOS states that DINOv2 class tokens are largely pose-invariant. That is good for recognizing identity, but it makes the view profile nearly flat and uninformative. FoundPose shows that intermediate patch descriptors (layer 18 of ViT-L), aggregated into bag-of-words template descriptors, keep orientation information. FoundPose also reports that its local refinement cannot recover from 180° flips on symmetric objects. We therefore reuse FoundPose's descriptors as $\phi$. Our profiles only need to recover the pattern of similarities across views, so they never commit to a single pose. Neither CNOS nor FoundPose rejects crops that match no catalogued object; both assume the object is known.

Symmetry handling supplies the group $G$ and a warning about it. EPOS handles symmetry with many-to-many surface-fragment correspondences but is strictly closed-world. BOP-Distrib computes elementary symmetry patterns from CAD models and shows that occlusion and viewpoint can create symmetries that the full model does not have. We reuse its symmetry-pattern computation to obtain $G$. We treat its occlusion finding as a threat to test, because a partially occluded asymmetric part may produce a "symmetric-looking" profile that a full-object prototype rejects.

Research Program

The program has three stages. First, we measure the bias. On all seven BOP core datasets we compute the CNOS score distributions for correctly matched known instances and for held-out objects. We then regress each object's mean genuine score and false-accept rate on its symmetry order: none, discrete two-fold, discrete higher-order, or continuous.

Second, we build the view-profile prototypes. We render roughly 600 icosphere views per object (FoundPose's sampling density), extract bag-of-words descriptors, and compute and symmetry-tie the self-similarity matrix offline. The cost is one $N\times N$ matrix per object, about 360k floats. The online cost of scoring a crop is one profile per candidate object, which reuses similarities the pipeline already computes. The only additional work is comparing profile shapes.

Third, we address occlusion-induced symmetry. We add masked prototypes: rows computed from templates whose patch features are restricted to the crop's visible-mask footprint. We then test whether this recovers recall on heavily occluded known parts without readmitting unknowns.

The hypothesis fails in either of two ways. If profiles of real crops do not resemble prototype rows more closely for known objects than for unknowns once domain gap is accounted for, the method has no signal. If per-object score calibration alone removes the bias, the extra machinery is unnecessary. Both outcomes are tested directly below.

Baselines

The naive reference point is CNOS with its own mean-of-top-5 score and a single global threshold. This is the pipeline as deployed. It reaches 41.2% mean average precision across the seven BOP core datasets in CNOS, compared with 39.2% for a supervised Mask R-CNN. That number measures segmentation of known objects only. No reported result was found on Lacuna for unknown-object rejection on any BOP dataset, so all rejection numbers in this proposal will be produced under our own protocol.

The strongest calibration competitor is CNOS with per-object thresholds. For each object, the threshold is set at a fixed true-positive rate on held-out renders and synthetic training images of that object. This directly cancels any symmetry-dependent shift in score level. Our method must beat it on false-positive rate at 95% true-positive rate (FPR95) to show that profile shape carries information beyond calibrated level.

The strongest learned competitor is an ADB-style head trained on the same frozen DINOv2 and bag-of-words features. It has one learned spherical radius per object, following Adaptive Decision Boundary. A variant uses one radius per view-cluster. Both are trained on the same synthetic renders, with the same budget for hyperparameter search.

The most serious competitor overall is render-and-verify. FoundPose estimates a coarse pose, the object is rendered at that pose, and the crop is accepted only if the featuremetric alignment error is low. This is expensive, but symmetric flips do not hurt it, because a flipped symmetric object renders identically.

Ablations isolate the key ingredients. We remove symmetry tying, swap FoundPose descriptors for class tokens, and replace standardized profile distance with the raw top-1 score. All methods share identical proposals from the Segment Anything Model, identical renders, and identical held-out splits. Beating the baselines means reducing FPR95 on lookalike unknowns relative to per-object-calibrated CNOS while keeping known-object average precision within one point of 41.2%.

Why not just…?

Why not just calibrate a threshold per object? This is the alternative most likely to absorb the gain. If symmetry only shifts each object's score level, per-object thresholds fix the problem completely. Our argument is that a lookalike unknown can reach a calibrated score level while producing the wrong profile shape, for example one sharp peak where a two-fold symmetric object should show two. The per-object baseline is in the main comparison. If it matches our method on the T-LESS lookalike split, we will report that the bias is purely a calibration effect and narrow the contribution to the diagnosis.

Why not just use top-1 similarity, or deduplicate symmetric templates? Top-1 removes the top-5 inflation but discards all relational information. It is also the most fragile score under the gap between rendered and real images, which is the reason CNOS averages the top five in the first place. Deduplicating templates by symmetry is exactly our symmetry-tying step without the profile comparison. Both are ablations, so their contribution is measured rather than assumed.

Why not just render at the estimated pose and verify? This is a real threat on accuracy. It costs a pose estimate (1.7 seconds coarse per FoundPose) plus a render for every proposal and every candidate object, whereas CNOS matching takes about 0.12 seconds. When the coarse pose is wrong, a known object is rejected. We will report accuracy and latency together. If render-and-verify dominates at matched latency, the right conclusion is that view profiles are a cheap pre-filter in front of it, and we will test that cascade explicitly.

Why not just train an open-set head or synthesize fake unknowns? Learned radii assume round class regions, which is the assumption under test, and the ADB-style baseline measures it directly. Synthesizing unknowns is the approach Reciprocal Points argues is unstable compared with structuring the feature space geometrically. More importantly, it would break the training-free onboarding that makes CAD-based pipelines attractive in the first place.

Proposed Experiments

The main protocol is leave-objects-out on the BOP core datasets. For each dataset we remove the CAD models of a random 30% of objects from the library over five seeds, so their instances become unknowns. T-LESS receives a curated "lookalike" split in which held-out objects have a near-duplicate kept in the library. The metrics are rejection AUROC and FPR95 over proposals, known-object segmentation average precision, and per-proposal latency. We stratify all metrics by the symmetry order of the matched known object and by visible fraction.

The first decisive check is the stratified analysis. If CNOS's false-accept rate rises with symmetry order and our method flattens that trend relative to per-object calibration, both parts of the core claim hold.

The second decisive check runs on YCB-V and LM-O, whose objects are more textured and include few near-duplicates. There the profile method should help less, which tests whether the gain is specific to symmetric, textureless parts rather than generic.

The third decisive check is occlusion-stratified recall on known objects. It shows whether the BOP-Distrib effect, where occlusion creates apparent symmetry, makes full-object prototypes reject valid occluded parts, and whether masked prototypes repair this.