Scale-Dependent Symmetry Labels for Coarse-to-Fine 3D Manipulation Policies

Many objects a robot handles are almost symmetric. A keyed peg is a cylinder except for a 3 mm notch. A connector is a square except for one chamfered corner. A mug is a ring except for its handle. When a policy looks at such an object coarsely, through a 1 cm voxel grid, a 512-point cloud, or the first stage of a coarse-to-fine search, the small feature that breaks the symmetry disappears. At that scale the object really is symmetric: several orientations are equally consistent with what the network can see. Current 3D policies still train every scale against the single orientation in the demonstration. We propose to do something different. The training target at each scale of a hierarchical policy should be the set of poses that cannot be told apart at that scale. The policy keeps those alternatives alive at coarse scales and commits to one only at the scale where the asymmetric feature becomes resolvable.

The opportunity is concrete. Policies that lift camera features into metric 3D get much of their robustness from coarse-to-fine designs. Act3D searches a workspace-wide grid, then a 16 cm ball, then a 4 cm ball (Act3D). PerAct classifies over a fixed grid of roughly 1 cm voxels (Perceiver-Actor). 3D Diffusion Policy (DP3) works from as few as 512 points (3D Diffusion Policy). None of these models treats object symmetry as something that depends on resolution. The pose-estimation literature, which does handle symmetry, treats it as a fixed property read off the object's mesh. If we are right, the failures these policies show on near-symmetric insertion should be mostly "flips" by exactly the hidden symmetry angle. A training-time change of labels should remove them, with no new sensors and no denser grids.

Core Claim

Define an object's approximate symmetry group $G$ as the rotations that map its mesh onto itself once features below some size are ignored. For a peg with one notch this is the full rotation group about the peg's axis. For each stage $s$ of a coarse-to-fine policy, define the scale-dependent ambiguity set

$$A_s = \{\, g \in G : d_s(o, g\cdot o) < \tau \,\}.$$

Here $o$ is the object's observation in its demonstrated pose, $g\cdot o$ is the observation with the object rotated by $g$, and $d_s$ is a distance between the two after both are reduced to the effective resolution of stage $s$ (voxel size, point budget, or ghost-point spacing). $\tau$ is a threshold set by sensor noise. At a coarse stage $A_s$ is large. At the stage where the notch spans enough samples, $A_s$ shrinks to the identity. Each stage is then trained with a set-valued loss:

$$\mathcal{L}_s = \min_{g \in A_s} \ell\big(\hat a_s,\; g \cdot a^*\big),$$

where $\hat a_s$ is the stage's predicted action (location heatmap and/or rotation) and $a^*$ is the demonstrated action.

The claim is that this loss, combined with carrying the top $k$ distinct hypotheses from one stage to the next, removes resolution-induced flip and averaging errors. It does so with no change at test time beyond a small beam. The mesh is needed only to compute labels during training.

Relation to Prior Work

Three lines of work motivate the gap without filling it.

Symmetry-aware pose estimation gives us the mechanism of failure and the form of the fix. Regression against one of several equally valid targets drives the network toward their average (3D Pose Regression Using CNNs; On Object Symmetries and 6D Pose Estimation). Minimizing over a symmetry group fixes this, as in PoseCNN's shape-matching loss (PoseCNN) and CFPA's best-aligned Chamfer loss for part assembly (CFPA). We reuse this min-over-set form. The difference is that all of these methods take the set from the mesh, once, at a single scale. For a notched peg that set is the identity, so they offer no relief exactly when downsampling makes the peg look symmetric. SymCode and Bingham-loss methods (SymCode; Bingham Loss) share the fixed-symmetry assumption and operate on 2D images.

BOP-Distrib comes closest (BOP-Distrib). It shows that ambiguity is specific to each image, depending on viewpoint and occlusion, and computes per-image pose distributions as ground truth for evaluation. Its authors name sensor resolution as an uncovered factor. We extend their idea in two ways: we let the ambiguity set depend on the network's own internal resolution rather than on mesh visibility, and we use it as a training label inside a policy rather than only as an evaluation target.

Coarse-to-fine 3D policies supply the architecture we modify. Act3D's hierarchy is our backbone. DP3 and PerAct are the regimes where the resolution loss is most extreme.

One honest constraint follows from how these policies predict location. Act3D scores ghost points by classification, not regression, so its location stage can already spread probability over a ring of valid grasp points. The averaging pathology should therefore show up mainly in the rotation head, and in cases where the single top coarse point centers a fine stage that never contains the notch. We will measure where errors actually arise rather than assume they come from the location stage.

Research Program

The first component is a label generator. For each simulated object we compute the approximate group $G$ by sweeping rotations of the mesh with features below a size cutoff removed. For each stage resolution we then render point clouds of the original object and rotated copies. We downsample each copy exactly as the policy's own pipeline does: the same voxel size, the same farthest-point sampling budget, or the same ghost-point spacing. We compare copies with Chamfer distance and keep the rotations whose distance falls below a noise-calibrated threshold. This produces $A_s$ for every training frame and stage, and it captures occlusion for free because we render from the actual camera. In simulation this costs only offline compute.

The second component is the training change. Each stage of Act3D is trained with $\mathcal{L}_s$. For the location heatmap this means the target is the union of the rotated contact points. For rotation it means the min-over-set loss on the rotation head attached to that stage. We add a small auxiliary head that predicts the size of $A_s$, so we can check whether the network knows when it is uncertain.

The third component is the inference change. We carry the top $k$ ghost-point hypotheses, separated by more than the stage radius, into the next stage, and let the fine stage choose among them. This takes $k$ times the cost of the finer stages, and we will report wall-clock time.

The fourth component is a diagnostic: the asymmetry-visibility curve. Error is plotted against the ratio of feature size to effective resolution, separately for "flip" errors (rotation error within a few degrees of a nontrivial element of $G$) and "blur" errors (rotation error lying between two elements, the signature of averaging).

Baselines

The naive reference is Act3D trained as published, with unimodal targets. It is the strongest existing method in the file on RLBench: 65% multi-task success, 22 points above PerAct, and 83% across 74 tasks (Act3D). On our new near-symmetric suite there is no reported result on Lacuna for any method, so every number there will be new. We retrain Act3D on the same demonstrations, with the same seeds and the same tuning budget, using its released hyperparameters. Beating it means higher success on near-symmetric tasks with no drop on the standard RLBench tasks.

The strongest symmetry competitor is Act3D with a shape-matching-style min-over-group loss, where the group is taken from the mesh at full resolution (PoseCNN). We also run a variant that declares near-symmetric objects fully symmetric. The first variant should give no benefit on notched objects. The second should destroy the fine-stage commitment. Our method must beat both on keyed insertion.

We include DP3 as the extreme-downsampling regime (3D Diffusion Policy). It reports 74.4% average success over 72 tasks. Here we apply the scale-dependent labels only to its 512- and 1024-point inputs, to test whether the label change carries over to a single-scale, diffusion-based policy.

Two ablations isolate the key ingredient: scale-dependent labels without hypothesis propagation, and $k$-hypothesis propagation with the original single-pose labels. An oracle that sets $A_s$ from ground truth at test time gives an upper bound. All variants share data, compute, and the same number of tuning trials.

Why not just…?

Why not just increase resolution? Dense grids grow with the cube of resolution, and PerAct already needs latent cross-attention to handle its $100^3$ grid (Perceiver-Actor). We still treat this as a real competitor. We run Act3D with a finer and denser final stage, using extra compute equal to what our $k$-beam costs. If that matches our method, the honest conclusion is that a better fine stage is all that is needed, and the visibility curve will tell us so.

Why not just let a diffusion or multi-hypothesis head handle the modes? This is the most serious threat. A generative head can represent several valid orientations without special labels. Two things count against relying on it. BOP-Distrib found that distribution-based pose estimators, including LiePose-Diffusion, still struggle with precision under ambiguity (BOP-Distrib). And a generative head trained on single demonstrations learns the demonstrators' habits, not the true set of valid poses. We settle this directly with a diffusion rotation head on Act3D and with DP3, each trained with and without our labels. If the plain diffusion versions show no flip or blur peaks, the main claim fails.

Why not just augment with rotations about the symmetry axis? For a notched peg, rotating the object also moves the notch. Augmentation therefore generates new unambiguous examples at fine scale and new conflicting labels at coarse scale, which is the averaging problem itself (On Object Symmetries). It also risks the harms of blind augmentation documented in To Augment or Not to Augment?. We include it as a baseline.

Proposed Experiments

The main testbed is a set of RLBench-style insertion and placement tasks with keyed pegs, chamfered blocks, and handled cups. Feature size is varied from 2 mm to 2 cm and camera distance in three steps, with 100 demonstrations per task as in THE COLOSSEUM (COLOSSEUM). We report success, rotation-error histograms, and visibility curves.

We then test robustness on COLOSSEUM's camera-pose and object-size perturbations, where 3D models already lose 30–50% of their success. The question is whether our gains survive those perturbations.

To show the label idea is not specific to policies, we run a pose-estimation check on T-LESS. We train a feature-pyramid pose network with scale-dependent sets at each pyramid level and evaluate with BOP-Distrib's per-image distributions, against the Pitteri et al. 41.27% VSD recall (On Object Symmetries).

The decisive check is the shape of the error distributions. If baselines show flip peaks that appear exactly when the notch falls below stage resolution, and those peaks vanish under our labels while compute-matched resolution scaling and plain diffusion heads leave them in place, the claim holds. If flips are already rare in vanilla Act3D because its 4 cm fine stage resolves the notch, the claim is falsified for continuous feature fields and survives only for the PerAct and DP3 regimes. We would report that outcome as such.