Meta-Learned Symmetry Labels at the Cost of One Forward Pass

When an object is symmetric, such as a bolt, a bowl or a four-sided industrial part, one image is consistent with several equally correct 3D orientations. Pose datasets nevertheless record a single orientation per image, chosen arbitrarily by the annotator. A network trained to regress that orientation gets contradictory targets for near-identical images and learns to predict their average. That average is often not a valid rotation at all (3D Pose Regression Using Convolutional Neural Networks, On Object Symmetries and 6D Pose Estimation from Images). We propose to treat the choice among equivalent labels as something the training procedure learns. Each training image keeps a soft weighting over its symmetric copies of the ground-truth rotation. The weighting is updated by asking which choice of copy most reduces error on a small held-out set whose loss counts every valid orientation as correct.

This is a two-level optimization problem, and such problems are normally expensive. Learning per-example weights by looking one step ahead roughly triples training time (Learning to Reweight Examples for Robust Deep Learning). Methods that sample more equivalent poses become steeply more costly as the sample count grows (Generalizable and Actionable Parts Pose Estimation with Symmetry Annotation-Free Learning Strategy). Our second claim is that, for this particular problem, the learning signal for every candidate label of every image in a batch can be obtained from a single forward-mode derivative pass. The cost therefore does not depend on how many symmetric copies are considered.

Core Claim

Training image $i$ has candidate rotations $R_{i1},\dots,R_{iK}$, obtained by applying the object's symmetry transformations to the annotated rotation. For continuous symmetries the candidates are $K$ samples around the symmetry axis. The image also carries weights $w_{ik}$ on the probability simplex. The training loss is

$$L_{\text{train}}(\theta, w) = \sum_i \sum_k w_{ik}\, \ell\big(f_\theta(x_i), R_{ik}\big),$$

where $f_\theta$ is the pose network and $\ell$ is an ordinary rotation regression loss. We take one look-ahead step of plain gradient descent with step size $\alpha$ and evaluate it on a validation loss $L_{\text{val}}$ that is invariant to symmetry. To first order, the effect of each weight is

$$\frac{\partial L_{\text{val}}\big(\theta - \alpha \nabla_\theta L_{\text{train}}\big)}{\partial w_{ik}} \approx -\alpha \,\big\langle \nabla_\theta L_{\text{val}}(\theta),\ \nabla_\theta \ell(f_\theta(x_i), R_{ik}) \big\rangle .$$

This is the gradient-alignment quantity identified by Learning to Reweight Examples for Robust Deep Learning. We use it for a different purpose. That paper uses the sign of the alignment to keep or drop whole examples. For a symmetric object, two equally valid labels of the same image can have opposing gradients, so its rule would discard correct data as noise. We instead redistribute weight within each image's set of symmetric copies. The total weight per image stays fixed, and the alignment only decides which copy carries it.

All $N \times K$ inner products are directional derivatives of the per-candidate losses along one vector, $v = \nabla_\theta L_{\text{val}}$. A single forward-mode pass (a Jacobian-vector product) through the training forward computation therefore returns all of them. This is the same move Scalable Meta-Learning via Mixed-Mode Differentiation makes when it replaces memory-heavy reverse-over-reverse products with forward-over-reverse ones. In our case it means no stored double-backprop graph and no cost that grows with $K$, because the $K$ candidates differ only at the output head.

The weights persist across epochs as per-image state and are updated multiplicatively. The hypothesis is that this procedure finds labelings that are consistent across nearby viewpoints. That would avoid the arbitrary seams that a fixed canonical mapping imposes, and the inconsistent early lock-in that a per-example "closest candidate" loss produces.

The held-out loss uses per-image sets of valid poses in the style of BOP-Distrib: Revisiting 6D Pose Estimation Benchmarks for Better Evaluation under Visual Ambiguities. Those sets are built from symmetry patterns of the visible surface, with occluded hypotheses pruned, so the loss credits any orientation that is truly indistinguishable in that image. This also gives a principled treatment of almost-symmetric objects. If a texture mark breaks the symmetry in some views, the held-out loss penalizes the wrong copies there, and the weights should concentrate on the true one. A loss that is invariant to the full symmetry group cannot learn this distinction.

Research Program

The method is built as a drop-in replacement for the symmetry handling in two existing regressors. The first is an RGB rotation regressor in the style of On Object Symmetries and 6D Pose Estimation from Images. Its fixed mapping of each label to a canonical rotation, together with its zone classifier, is removed and replaced by learned weights. The second is the RGB-D regressor of ES6D: A Computation Efficient and Symmetry-Aware 6D Pose Regression Framework, whose grouped-primitive loss we replace or combine with ours.

For each training batch, the procedure has four steps. We compute the held-out gradient on a small validation batch. We run one forward-mode pass of the training batch along that gradient to obtain all alignment scores. We update the weights. We then take an ordinary training step under the updated weights.

The analysis tracks three questions. The first is labeling consistency: how often the chosen copy jumps between images whose true orientations are close. The second is wall-clock time and peak memory relative to plain training. The third is how the weights behave on near-symmetric versus exactly symmetric objects.

A secondary study asks whether the one-step first-order signal is enough. We replace it with a three-step unrolled look-ahead, made affordable with the mixed-mode differentiation of MixFlow-MG. If the longer horizon improves consistency, the extra second-order cost is justified and we quantify it. If it does not, the cheap version stands. This is the point where the second-order overhead question and the symmetry question meet directly.

Baselines

The naive reference point is plain regression against the single annotated rotation. It shows how much mean collapse costs on each benchmark.

The most direct competitor is the per-example minimum over symmetric copies, that is, a loss like ShapeMatch from PoseCNN: A Convolutional Neural Network for 6D Object Pose Estimation in Cluttered Scenes. Its cost is negligible and it already makes the loss zero for every valid rotation, so we must beat it to justify the method. A soft-minimum version, with a temperature annealed over training, is included as well. It uses the same weight parameterization as our method, but the weights are set from the current prediction rather than from the held-out signal. This version is the ablation that isolates our key ingredient: the two-level selection of labels. Any gain over it must come from looking ahead to validation error, not from soft weighting.

The fixed canonical mapping with zone classifier of On Object Symmetries and 6D Pose Estimation from Images represents global hand-designed labelings. On T-LESS it reports 41.27% mean VSD (visible surface discrepancy) recall. The strongest symmetry-aware regressor in the file with a reported T-LESS figure is ES6D, which reports a VSD score of 0.76 versus 0.73 for the next best method. GCPose reports 67.9% Average Recall on T-LESS (Learning Symmetry-Aware Geometry Correspondences for 6D Object Pose Estimation). It uses correspondence matching with depth and CAD models, so we report it as context rather than as a matched comparison.

The original example-reweighting procedure of Learning to Reweight Examples for Robust Deep Learning is run unchanged to test whether it indeed discards valid symmetric data. A probabilistic output head with the Bingham loss of Deep Orientation Uncertainty Learning based on a Bingham Loss covers the alternative of modeling ambiguity in the output rather than in the labels.

All methods share the same backbone, training data, augmentation, epochs and hyperparameter search budget. Our method takes its held-out batch from images that are removed from everyone's training set. Beating the baseline means improving VSD and BOP-Distrib scores over the soft-minimum ablation and over the reproduced ES6D at matched wall-clock time, with the gain concentrated on symmetric and near-symmetric objects.

Why not just…?

Why not just use a symmetry-invariant minimum loss? This is the real threat. It is cheap, and it is the default choice. Its weakness is that each image's choice of copy depends on the network's current guess. Early, arbitrary choices can therefore become inconsistent across viewpoints, leaving a discontinuous target the network must fit. ES6D reports that ADD-S-style losses have false local minima (ES6D: A Computation Efficient and Symmetry-Aware 6D Pose Regression Framework). The minimum loss also treats near-symmetric objects as exactly symmetric. The soft-minimum ablation, paired with the labeling-consistency measurement, settles the question. If the minimum loss already yields consistent labelings and equal accuracy, our method adds nothing, and we will report that.

Why not just canonicalize labels once, offline? A fixed canonical mapping creates seams at zone boundaries, and On Object Symmetries and 6D Pose Estimation from Images needs an extra classifier plus several regressors to handle them. Symmetric surface codes need manual offline partitioning of the CAD model (Resolving Symmetry Ambiguity in Correspondence-based Methods for Instance-level Object Pose Estimation). Neither adapts to symmetry that is broken in some views. We test this directly by comparing error as a function of distance to the canonical seams.

Why not just predict a distribution over poses? Bingham or diffusion outputs represent ambiguity well (DiffusionNOCS: Managing Symmetry and Uncertainty in Sim2Real Multi-Modal Category-level Pose Estimation reports 35.0 mAP at 5°5cm on NOCS Real 275). However, the Bingham loss needs a fragile two-stage training scheme, and diffusion needs many samples plus a registration step at inference. Our method is complementary because it cleans the labels any output head sees. We test it both with a point-estimate head and combined with the Bingham head.

Proposed Experiments

The primary benchmark is T-LESS, which consists of thirty textureless and largely symmetric industrial parts. It is evaluated with standard BOP VSD recall and with BOP-Distrib distribution-aware scores, since single-pose scoring can misrank methods under ambiguity. YCB-Video, with five symmetric classes, tests whether the method harms asymmetric objects. GAPartNet tests settings where symmetry type is uncertain. There, SAFAG reports 3.23° average rotation error on seen categories and 10.83° on unseen ones, and the question is whether learned weights over a superset of candidate copies match it without being told the symmetry type.

Cost is measured as wall-clock time per epoch and peak memory. We compare plain training, the original look-ahead reweighting (reported at roughly three times plain SGD), double-backprop on our objective, and our forward-mode version. Each is measured at $K \in \{4, 16, 64\}$ to show that our cost stays flat while sampling-based approaches grow.

Three checks decide the question. The first is whether learned weights produce lower labeling inconsistency between neighboring viewpoints than the soft-minimum ablation. The second is whether that consistency translates into VSD gains on symmetric T-LESS objects at no more than about 1.3 times plain training time. The third is whether, on near-symmetric objects, the weights concentrate on the true copy in views where the breaking feature is visible.

If the first check fails, the hypothesized mechanism is wrong, and we would not attribute any accuracy gain to it. If the second fails while the first holds, consistency is not the bottleneck on these benchmarks.