Robots that pick up mugs, cameras or laptops need to know each object's full 3D position and orientation in every video frame. Current trackers for whole object categories, which work on new instances without a 3D model of each one, are fast and accurate once running, but each must be given a correct starting pose. A natural fix is to start them with a single-frame pose estimator. That fix fails in a specific and common situation. When tracking begins, the feature that tells you which way the object faces, such as a mug's handle or a camera's lens, is often hidden. The starting pose is then a guess, and the tracker keeps that guess forever, because it only predicts small frame-to-frame corrections around its current belief.
We propose a tracker that starts itself with a set of weighted orientation hypotheses instead of one answer. It follows all of them cheaply and drops the wrong ones only when evidence from the color image shows the symmetry-breaking part. In short, the tracker holds a symmetric belief while the object looks symmetric, and it uses camera-image semantics to settle the belief when the object stops looking symmetric. The question we test is whether postponing the decision about orientation, rather than adding modalities or capacity, is what makes self-initialized category-level tracking reliable.
Core Claim
Self-initialization and symmetry ambiguity are the same problem viewed at two moments in time. An object's symmetry is not fixed by its shape. BOP-Distrib shows that an asymmetric object such as a mug becomes visually symmetric when its distinguishing part is occluded, so the set of valid poses changes from image to image. A tracker that starts on such a frame must represent that set, and it must shrink it later when the evidence arrives. We claim that a tracker which carries a small mixture of rotation hypotheses in the category's shared coordinate frame will beat commit-at-start pipelines on sequences where the decisive part is hidden at the start. The mixture is reweighted by appearance features lifted from the RGB image onto the depth points. We also claim it will match them on sequences where nothing is hidden, at comparable frame rate. The new element is not multi-modal input or uncertainty on its own. It is the online, appearance-gated collapse of a symmetry-shaped belief inside a category-level tracking loop.
Where Prior Work Stands
The tracking side supplies our machinery and also motivates the gap. CAPTRA tracks category-level pose from point clouds by using the previous pose to move each new frame into a near-canonical frame and then regressing a small residual rotation. It runs at about 12 FPS but assumes a known first-frame pose, and it ignores RGB. KPA-Tracker uses the same canonicalize-then-regress-increments design for articulated objects and has the same starting-pose requirement. BundleTrack needs no model but needs a first-frame mask, and it drifts on smooth symmetric objects where keypoints are unstable. We reuse CAPTRA's canonicalization as the per-hypothesis update. What we change is that there are several canonicalizations in parallel, weighted by evidence.
The single-frame side supplies the starting hypotheses. NOCS predicts dense coordinates in a shared per-category frame and handles symmetric categories with a min-over-symmetries loss. SSP-Pose regresses pose directly at 25 FPS with symmetry-aware candidate losses. Both output one pose. DiffusionNOCS conditions a diffusion model (a generator that produces samples by iteratively denoising noise) on RGB, depth normals and DINOv2 features. It produces multiple valid coordinate maps for symmetric objects, but it is slow, works frame by frame, and selects one hypothesis by registration inlier rate. Because inlier rate is purely geometric, it cannot tell symmetric hypotheses apart. Deep Bingham Networks and the Bingham loss predict mixtures of Bingham distributions, which are probability distributions on unit quaternions that respect $q \equiv -q$. They are trained with a relaxed winner-takes-all loss so that each component captures one valid orientation. We take this output head, but these works are single-frame and instance-level. The authors also note that the Bingham normalizing constant needs lookup tables and becomes unstable when uncertainty is high. For that reason we use the mixture only to spawn and weight a few discrete tracks, not to integrate densities in the loop.
The semantic side motivates the appearance gate and also constrains it. ConceptPose shows that language-derived part concepts such as "handle" and "rim", lifted to 3D, break visual symmetries when the parts are visible. It reports 71.5% ADD(-S) on REAL275 zero-shot, but it takes about 7 seconds per frame and fails when the concept parts are occluded. That failure is exactly the situation our tracker is designed to wait out. SOCS shows that canonical frames should keep semantic parts in consistent places across instances, but its warping breaks rotational symmetry axes. We therefore keep the rigid NOCS frame and put part semantics in features rather than in geometry. One constraint cuts against us. GCPose reaches 67.9% AR on T-LESS by deliberately discarding RGB, arguing that appearance hurts generalization. Our design takes that seriously by using appearance only to reweight hypotheses that geometry already proposes, never to propose poses. The ablations test whether this gating keeps the generalization benefit of a geometry-only method.
Research Program
The system has three parts.
The first is a starter. On any frame, the starter takes an RGB-D crop from an open-vocabulary category detector and outputs a $K$-component Bingham mixture over canonical rotation, plus translation and size. Its point features are formed by early point-wise fusion in the style of PointFusion: each depth point is projected into the image and concatenated with the frozen DINOv2 feature at that pixel. Training uses the relaxed winner-takes-all loss on CAMERA synthetic data and the REAL275 training split. The intended effect is that on a handle-hidden mug the components spread around the vertical axis, and on a handle-visible mug a single component dominates.
The second part is a per-hypothesis tracker. Each surviving component $k$ becomes a track that is updated by a CAPTRA-style residual network shared across tracks. With $K \le 8$ and batched inference, we target at least 10 FPS.
The third part is an evidence gate. For each track $k$ at frame $t$, a small network compares the fused point features with a learned category-level part map defined in canonical coordinates. It produces a log-likelihood $\ell_{k,t}$ and a visibility score $v_t \in [0,1]$, which estimates whether any symmetry-breaking part is in view. Weights are updated as
$$ w_{k,t} \propto w_{k,t-1}\,\exp\!\big(v_t\,\ell_{k,t}\big), $$
where $w_{k,t}$ is the weight of hypothesis $k$ at frame $t$. The update is a Bayesian filter step whose evidence term is softened by $v_t$. When nothing distinguishing is visible ($v_t \approx 0$), weights do not move, which is the "wait" behavior. Tracks whose weight falls below a threshold are pruned. A track that loses its geometric fit triggers the starter again, which handles occlusion-induced loss and objects entering the scene.
The gate is trained on synthetic sequences in which the visibility of parts is known, using a weight cross-entropy against the correct mode. The training also includes an explicit penalty on weight changes during frames where geometry and appearance cannot distinguish the hypotheses. This penalty targets the failure mode we most fear: a gate that confidently collapses on texture noise.
Baselines
The naive reference is a tracker given no warm start. CAPTRA is started from an identity rotation, with translation and size taken from the detection box. This shows how much any initialization matters.
The practical decoupled baseline is SSP-Pose run on frame 0, its single pose handed to CAPTRA. SSP-Pose is the fastest strong single-frame estimator in the file, reporting 82.3% IoU50 and 34.7% at 5°2cm on REAL275 according to SSP-Pose. This is the system a practitioner would actually build. We retrain both components on the same CAMERA plus REAL275 training data as our method, with the same detector, the same GPU, and an equal hyperparameter search budget.
The strongest existing tracking number is BundleTrack's 87.4% at 5°5cm on REAL275 at 10 Hz, from BundleTrack. It is obtained with a first-frame mask under its own protocol. We will run it both with its oracle mask and with our detector's mask, and we will report it as a ceiling-like reference rather than a like-for-like competitor. CAPTRA's own result is reported only as a 40.03% absolute gain over 6-PACK at 5°5cm, so no absolute CAPTRA number is available in the file and we will measure it ourselves under its ground-truth-start protocol as an oracle upper bound.
Two baselines isolate our ingredients. The first is parallel trackers with geometric selection: $K$ CAPTRA tracks started from DiffusionNOCS samples, or from our starter's components, and pruned by registration inlier rate in the way DiffusionNOCS selects hypotheses. This isolates the appearance gate. The second is a one-hypothesis version of our full system, with fusion and gate but $K=1$. This isolates deferred commitment.
Beating the field means two things. On the handle-hidden-at-start subset defined below, we must beat the SSP-Pose→CAPTRA pipeline and the parallel-geometric baseline in 5°5cm by a margin larger than the seed variance. On the full test set, we must stay within noise of the oracle-started CAPTRA.
Why not just…?
Why not just start CAPTRA with a good single-frame estimator? When the decisive part is visible at frame 0, this probably works as well as our method, and we expect parity there. The failure is structural. CAPTRA regresses residuals near identity, so a starting rotation that is wrong by up to 180° about the mug's axis is never corrected. The handle-hidden subset is designed to expose this, and the "wrong-mode lock-in rate" metric measures it directly. If lock-in turns out to be rare on real data, our claim shrinks to an edge case, and we will report that.
Why not just rerun a single-frame estimator on every frame? Tracking-by-detection does pick up the handle once it appears, so it is a real threat. It has no memory, however, so it flips orientation again whenever the handle disappears, and it inherits per-frame detector noise. STIC documents a drop of more than 10% AUC on LaSOT from detector-quality boxes in 2D tracking. We will run SSP-Pose every frame with temporal smoothing and measure 5°5cm, frame-to-frame flip rate, and jitter. If it matches us on all three, the tracking machinery is unnecessary.
Why not just run several hypotheses in parallel and keep the best geometric fit? For a truly symmetric object, geometry scores all modes equally, so geometric selection is effectively random among them. Visible semantic parts are what break the tie, which is what ConceptPose exploits, though far too slowly for tracking. The parallel-geometric baseline tests this claim directly. If it matches the full method, the gate adds nothing, and the contribution reduces to multi-hypothesis starting.
Why not just add DINOv2 features to CAPTRA? Fusion without deferral still commits at frame 0. Fusing dense image tokens with sparse geometry is also known to let texture dominate, as studied in Countering Multi-modal Representation Collapse. The $K=1$ ablation is exactly this model.
Proposed Experiments
The main benchmark is the NOCS-REAL275 test videos, using its six categories, where mug and camera carry occludable symmetry-breaking parts and bottle, bowl and can are axially symmetric. From ground-truth poses and object models, we label for every frame whether the handle or lens is visible. We then build a "hidden-start" protocol in which every tracker is started at frames where that part is invisible, alongside the standard full-sequence protocol. We report 5°5cm, 5°2cm and IoU50, together with two diagnostics. Lock-in rate is the fraction of tracks still in the wrong rotational mode 50 frames after the part becomes visible. Time-to-resolution is the number of frames from visibility to the correct mode reaching weight above 0.9.
YCBInEOAT tests transfer to robot-manipulation sequences, where the hand repeatedly occludes the object. There we report ADD-AUC; ConceptPose reports 90.1% ADD-AUC in few-shot tracking, but on YCB-Video rather than YCBInEOAT, so that number is not directly comparable. To check that the starter's mixture is calibrated rather than merely multi-headed, we evaluate it on T-LESS with BOP-Distrib's per-image distribution ground truth and compare it to the single-pose SSP-Pose head. Runtime is measured on one GPU for all systems.
Three checks would settle the claims. If the full method fails to beat SSP-Pose→CAPTRA on hidden-start lock-in rate, deferral does not matter and the idea fails. If it fails to beat the parallel-geometric baseline, the appearance gate is unnecessary. If it falls below oracle-started CAPTRA on the full protocol, or drops below 10 FPS, the method is not a practical replacement. A final stress test replaces DINOv2 features with random features of the same dimension. If the gate's weights still move on handle-hidden frames, the visibility score is not doing its job, and the reported gains would not be trustworthy.
References
- BOP-Distrib: Revisiting 6D Pose Estimation Benchmarks for Better Evaluation under Visual Ambiguities
- CAPTRA: CAtegory-level Pose Tracking for Rigid and Articulated Objects from Point Clouds
- KPA-Tracker: Towards Robust and Real-Time Category-Level Articulated Object 6D Pose Tracking
- BundleTrack: 6D Pose Tracking for Novel Objects without Instance or Category-Level 3D Models
- Normalized Object Coordinate Space for Category-Level 6D Object Pose and Size Estimation.
- SSP-Pose: Symmetry-Aware Shape Prior Deformation for Direct Category-Level Object Pose Estimation
- DiffusionNOCS: Managing Symmetry and Uncertainty in Sim2Real Multi-Modal Category-level Pose Estimation
- Deep Bingham Networks: Dealing with Uncertainty and Ambiguity in Pose Estimation
- Deep Orientation Uncertainty Learning based on a Bingham Loss
- ConceptPose: Training-Free Zero-Shot Object Pose Estimation using Concept Vectors
- SOCS: Semantically-aware Object Coordinate Space for Category-Level 6D Object Pose Estimation under Large Shape Variations
- Learning Symmetry-Aware Geometry Correspondences for 6D Object Pose Estimation
- PointFusion: Deep Sensor Fusion for 3D Bounding Box Estimation
- Spatial-temporal initialization dilemma: towards realistic visual tracking
- Countering Multi-modal Representation Collapse through Rank-targeted Fusion