Many real objects look the same after certain rotations. A mug without its handle in view, a cylinder, or a box turned 180° all produce the same image, so a camera cannot tell which of several orientations is the true one. Methods that estimate an object's 3D position and orientation ("6D pose") from single images have learned to live with this. They use losses that do not penalize a symmetric alternative, or they output a probability distribution over all valid poses. When these single-frame methods are run on every frame of a video, though, a new failure appears. Each frame is free to pick a different symmetric answer, so the predicted trajectory jumps by 90° or 180° between frames even though the object barely moved. Parallel sequence generators in machine translation show exactly this pathology: a decoder that outputs every word at once can write half of one valid translation and half of another ("Vielen Danke").
Our central claim is that these are the same problem, and that the fix from sequence generation carries over once one change is made. Translation systems remove the ambiguity by distilling an autoregressive teacher, which keeps one mode and throws away the others. Pose tracking should instead supervise the whole window with a single shared symmetry choice, which leaves the ambiguity intact. We propose a bidirectional transformer that predicts all poses in a video window in one pass. Its loss minimizes over one symmetry element for the whole window rather than one per frame, and it outputs a small set of whole-trajectory hypotheses rather than per-frame marginals. The expected outcome is coherent trajectories with no symmetry flips. Distinguishing evidence seen in any frame, such as a handle coming into view, should propagate to every other frame, and the method should keep all hypotheses that remain valid. It should also run faster than per-frame distribution estimators followed by temporal smoothing.
Core Claim
Let $R^*_{1:T}$ be the ground-truth rotations of an object over a window of $T$ frames. Let $G_W$ be the set of object-frame symmetries that are consistent with what is visible across the whole window. We compute $G_W$ by intersecting the per-image ambiguity sets produced by the procedure of BOP-Distrib, which builds ground-truth pose distributions per image from symmetry patterns of the visible 3D points and depth-based occlusion pruning. The set of equally correct answers is then the family of trajectories $\{R^*_{1:T}\,g : g \in G_W\}$. This is a set of whole trajectories, not a product of per-frame sets.
The model has $K$ hypothesis queries. Query $k$ emits a trajectory $\hat R^{k}_{1:T}$, where $\hat R^k_t$ is the rotation that query predicts at frame $t$. We train with
$$\mathcal{L} = \min_{k \le K}\; \min_{g \in G_W}\; \sum_{t=1}^{T} d\big(\hat R^{k}_t,\; R^*_t\, g\big) \;+\; \lambda\,\mathcal{L}_{\text{spread}},$$
where $d$ is the mesh-vertex alignment distance of PoseCNN's ShapeMatch loss (or the geodesic angle), and $\mathcal{L}_{\text{spread}}$ is a term that pushes the $K$ queries toward distinct elements of $G_W$ with weight $\lambda$. The minimum over $g$ sits outside the sum over frames, and that placement is the key ingredient. A per-frame symmetric loss places the minimum inside the sum, which lets the network pick a different $g$ at every frame and gives no gradient against flips. For discrete groups, which typically have at most 24 elements, the inner minimum is enumerated exactly. For continuous symmetry about a known axis, it reduces to a one-dimensional angle alignment with a closed-form solution. Translation is supervised directly, since the symmetries considered here do not affect it.
Research Program
The work builds on three lines of prior research, each playing a distinct role.
The first line motivates the problem. Single-frame symmetry handling is mature: Pitteri et al.'s map operator maps each rotation to a canonical representative and splits rotation space into zones, each with its own regression head. PoseCNN supplies the symmetry-blind ShapeMatch loss. The Bingham loss and SpyroPose output full orientation distributions, the second as a coarse-to-fine grid over $SE(3)$. All four operate per image. Pitteri et al. report that canonicalization creates seams where a small physical motion makes the canonical label jump. Over a video, an object rotating across a seam therefore produces exactly the flip we want to remove. Per-frame canonicalization is effectively per-frame distillation: it picks one arbitrary mode per frame, independently.
The second line supplies the methods. From non-autoregressive generation, we take the diagnosis from Gu et al. that independent parallel outputs mix modes. We also take the structured remedy of Sun et al., a linear-chain conditional random field (a model that scores whole output sequences using pairwise compatibilities between neighboring positions) placed on top of a parallel decoder. Our main design replaces distillation with window-level supervision. The random field serves as the main structured competitor, described under Baselines below.
The third line comes closest to our work, and we differ from each piece. POTR decodes future human poses in parallel with a 16x speedup over its autoregressive variant. It regresses unconstrained coordinates, has no symmetry handling, and averages modes; its authors name multi-modal futures as open work. PECCO builds planar rotation equivariance into trajectory forecasting, meaning its predictions rotate consistently when the input scene is rotated. It is restricted to planar rotation, assumes a single Gaussian mode, and uses recurrence. DiffusionReg denoises a single rigid transform on the manifold of 3D rigid motions and does not model sequences. Imitative NAR modeling decodes 2D Euclidean trajectories in parallel with no orientation. None of these methods treats symmetry ambiguity as sequence-level multimodality with a shared gauge, meaning one symmetry choice that holds for the whole window.
We will build a shared image backbone that processes $T$ object crops (window length $T$ of 16 or 32 frames) into per-frame tokens. A bidirectional transformer lets every frame attend to every other. $K$ trajectory queries, with $K$ set to the largest discrete group size in the dataset (capped at 8), decode rotations in a continuous 6D representation projected to valid rotations, together with translations. Every query receives a mode confidence. Continuous symmetries are represented by one query per mode, carrying an explicit free-angle flag instead of an enumerated set.
The analysis will measure three things. The first is whether evidence propagates: when a distinguishing feature appears only at frame $t_0$, do earlier frames collapse onto the correct hypothesis? The second is whether ambiguity is preserved: when no such feature is ever visible, do the $K$ queries cover $G_W$? The third is how flip rate varies with how often the object crosses the seams of Pitteri et al.'s canonical zones.
Baselines
The naive reference point is a per-frame estimator with the same backbone, trained with PoseCNN's per-frame ShapeMatch loss and run independently on each frame. PoseCNN reports 78.0% mean accuracy on Occluded-LINEMOD with depth refinement by iterative closest point, the standard point-cloud alignment step (PoseCNN). This baseline is symmetry-correct per frame and makes no attempt at temporal consistency, so it should show the full flip rate. A second naive reference replaces ShapeMatch with Pitteri et al.'s canonicalized zone targets. Their paper reports 41.27% mean recall on T-LESS under the Visible Surface Discrepancy metric (Pitteri et al.). This reference tests whether seams cause flips over time.
The strongest competitor is SpyroPose run per frame, followed by exact Viterbi smoothing over time. Viterbi is the standard dynamic-programming algorithm that finds the single most likely path through a chain of discrete choices. Here the per-frame states are SpyroPose's top modes, and the transition scores penalize the geodesic change between consecutive frames. We also run the same smoothing on LiePose-Diffusion outputs where they are available, since BOP-Distrib reports that it generally outperformed SpyroPose. This pipeline is the strongest competitor because it already represents per-frame multimodality well and adds the obvious temporal structure. SpyroPose runs at 16.1 FPS for full rigid-motion estimation on GPU (SpyroPose). We found no reported result on Lacuna for any sequence-level or distribution-based score of SpyroPose on T-LESS or YCB-V, so we will measure it ourselves under our protocol.
A structured parallel baseline follows Sun et al. It uses our transformer but has each frame emit logits over a shared candidate set, which is the base rotation composed with each element of $G$. A linear-chain random field is placed over these, with transition scores learned from hidden states and decoded exactly, since the small number of states makes this cheap. An autoregressive baseline uses a causal transformer that decodes frame by frame with the same backbone and the same window-level loss. This isolates the benefit of parallel decoding from that of the loss.
The key ablation is our model with the per-frame minimum (inside the sum) instead of the window-level one, with everything else identical.
All methods share the backbone initialization, training data, crops, and synthetic augmentation. Each receives the same GPU-hour budget and the same number of hyperparameter trials. Beating the strongest baseline means two things. The adjacent-frame flip rate must be at least halved, with no loss in per-frame BOP-Distrib precision and recall. And at equal batch size the method must process more frames per second than smoothed SpyroPose.
Why not just…?
Why not just smooth per-frame distributions with Viterbi? This is the real threat. If per-frame distributions are well calibrated, Viterbi recovers the most consistent path, and evidence from an unambiguous frame propagates through the transitions. Our bet is that per-frame estimators are miscalibrated in exactly the frames that matter. BOP-Distrib shows that ambiguity depends on occlusion and viewpoint per image, and Gilitschenski et al. report that multi-modal mixture training is fragile. A per-frame model therefore weights modes poorly when a distinguishing feature is half-occluded, while joint attention can pool weak evidence across frames. Viterbi also returns one path, so returning all of $G_W$ requires k-best decoding with extra bookkeeping. The occlusion-reveal experiment below settles this question. If smoothed SpyroPose matches us there, we will report that post-hoc structure is enough. The remaining contributions would then be the window-level evaluation protocol and the latency comparison.
Why not just decode autoregressively, conditioning each pose on the previous one? Causal decoding does remove flips going forward in time, but it cannot let a feature revealed at frame 20 disambiguate frame 3. POTR also reports a 16x slowdown and error drift for its autoregressive variant (POTR). The causal baseline, trained with the same window-level loss, measures both effects directly. We expect it to match us on forward consistency and lose on backward disambiguation and speed.
Why not just distill a teacher or canonicalize labels, as in non-autoregressive translation? Distillation keeps one mode per sequence. That is acceptable in translation (Gu et al.) but wrong for grasping, where every valid symmetric orientation matters, and BOP-Distrib shows that single-mode outputs are penalized under distribution metrics. Per-frame canonicalization is worse, because its seams produce flips. The Pitteri-target baseline and the coverage metric will quantify both effects.
Why not just add temporal attention to a per-frame symmetric loss? Attention alone gives the model no reason to agree across frames, since the per-frame minimum rewards any frame-wise mix of symmetric answers. The inside-versus-outside ablation tests exactly this claim, and it is the single most important comparison in the proposal.
Proposed Experiments
We use four evaluation settings. The first is YCB-V video sequences, which contain symmetric bowls, blocks, and clamps. The second is T-LESS scenes ordered by camera viewpoint, whose textureless industrial parts have both discrete and continuous symmetry. Both come with BOP-Distrib per-image distributions, which we intersect over windows to obtain $G_W$. The third is a controlled synthetic set: SYMSOL-style solids rendered along smooth camera paths, with a single asymmetric marking that is scheduled to appear, disappear, or never appear. This setting lets us control exactly where the ambiguity collapses.
The metrics are as follows. Per-frame BOP-Distrib precision and recall check that we do not trade accuracy for consistency. Flip rate is the fraction of adjacent frame pairs whose predictions differ by a nontrivial symmetry element beyond the true relative motion. Trajectory-set recall is the fraction of $G_W$ covered by at least one hypothesis within a threshold. Propagation accuracy is the correctness of frames before the reveal point in marked sequences. We also report frames per second at a fixed batch size.
Three results would decide the question. First, the inside-versus-outside ablation must show that the window-level loss alone cuts flip rate. Second, on the synthetic reveal sequences and on YCB-V frames annotated as partially occluded, we must beat smoothed SpyroPose on pre-reveal accuracy while matching it on unambiguous frames. Third, on never-revealed sequences, trajectory-set recall must match or exceed k-best Viterbi, which shows that the method keeps symmetric alternatives rather than dropping them. If the first check fails, the idea is wrong. If the first passes but the second does not, post-hoc smoothing is sufficient and we will say so.
References
- BOP-Distrib: Revisiting 6D Pose Estimation Benchmarks for Better Evaluation under Visual Ambiguities
- On Object Symmetries and 6D Pose Estimation from Images
- PoseCNN: A Convolutional Neural Network for 6D Object Pose Estimation in Cluttered Scenes
- Deep Orientation Uncertainty Learning based on a Bingham Loss
- SpyroPose: SE(3) Pyramids for Object Pose Distribution Estimation
- Non-Autoregressive Neural Machine Translation
- Fast Structured Decoding for Sequence Models
- Pose Transformers (POTR): Human Motion Prediction with Non-Autoregressive Transformers
- Probabilistic Symmetry for Improved Trajectory Forecasting
- SE(3) Diffusion Model-based Point Cloud Registration for Robust 6D Object Pose Estimation
- Imitative Non-Autoregressive Modeling for Trajectory Forecasting and Imputation