Symmetry-Aware Learning-Rate Control for Pose Estimators Under Gradual Drift

When a 6D pose estimator runs on a live camera stream, the world usually shifts slowly: lighting changes over a day, a lens gathers dust, a sensor warms up. Each frame looks almost the same as the last, so detectors that watch for abrupt changes never fire. Meanwhile the model quietly gets worse. Theory says the fix is to keep the learning rate of the online adapter proportional to how fast the data distribution is moving. This proposal targets a specific way that fix breaks for rotationally symmetric objects such as bowls, bottles, cans, and many industrial parts. For these objects, the model's prediction can jump between rotations that look identical in the image. A drift signal computed in ordinary rotation or parameter coordinates reads these harmless jumps as fast drift. It then spikes the learning rate exactly when the pseudo-labels (the model's own predictions used as training targets) are least trustworthy.

The idea is easy to state. We estimate drift velocity from the disagreement between a fast-adapting student model and a slow moving-average teacher. That disagreement is measured on the space of rotations modulo the object's symmetry group, so symmetry-equivalent predictions count as zero disagreement. The measured velocity then sets the step size of test-time adaptation. The hypothesis is that this makes the learning rate rise under real drift and stay flat under symmetry flips. A controller in plain Euclidean or full-rotation-group coordinates cannot do both.

Core Claim

Learning Rate Schedules in the Presence of Distribution Shift derives, for drifting streams, that the optimal step size should plateau or grow with drift velocity rather than decay. It estimates that velocity with an exponential moving average of parameter displacement. This paper supplies our control law. It works in flat parameter space and says nothing about outputs that live on a curved space with symmetries.

The symmetry literature shows why that omission matters. On Object Symmetries and 6D Pose Estimation from Images and 3D Pose Regression Using Convolutional Neural Networks show that identical images paired with different but equivalent rotation labels produce opposing gradients, which leads to averaging collapse and 180° flips. This is the mechanism that would contaminate a displacement-based drift proxy. Parameters move substantially while the physical estimate has not changed at all.

Our claim is that measuring drift in function space on the quotient manifold $SO(3)/G$, where $G$ is the object's symmetry group, gives a velocity estimate that is both task-relevant and blind to relabeling. The learning-rate controller built on it should beat both fixed-rate and Euclidean-velocity controllers, and the margin should concentrate on symmetric categories.

For two rotations $R_1, R_2$ the symmetry-quotiented distance is

$$d_G(R_1,R_2)=\min_{g\in G}\ \angle\!\left(R_1,\;R_2\,g\right),$$

where $\angle$ is the geodesic rotation angle. For continuous axial symmetry (bottle, can, bowl), this reduces in closed form to the angle between the predicted symmetry axes. For discrete groups it is a small minimum. The per-frame drift signal is $D_t = d_G(R^{s}_t, R^{T}_t)$, the disagreement between student prediction $R^s_t$ and teacher prediction $R^T_t$ on the same frame. We normalize it by a noise floor $\bar D^{\mathrm{shuf}}_t$ computed on a temporally shuffled buffer, following the chronological-versus-shuffled comparison of Concept Drift Detection Through Resampling. The step size is

$$\eta_t=\mathrm{clip}\!\left(\eta_0+\kappa\cdot \mathrm{EMA}\big[(D_t-\bar D^{\mathrm{shuf}}_t)_+\big],\ \eta_{\min},\ \eta_{\max}\right),$$

with base rate $\eta_0$, gain $\kappa$, and bounds $\eta_{\min},\eta_{\max}$. The rationale for this form is that a student tracking a moving optimum lags it by an amount roughly proportional to drift velocity. Student–teacher disagreement is therefore a function-space stand-in for the displacement proxy of Fahrbach et al. The shuffled floor removes disagreement caused by ordinary frame-to-frame noise rather than by trend.

Research Program

The substrate is TTA-COPE (TTA-COPE: Test-Time Adaptation for Category-Level Object Pose Estimation). It adapts a category-level pose network online from synthetic CAMERA to real REAL275 using a student and a moving-average teacher, a pose ensemble, and a depth-based inlier check, all at a fixed learning rate. TTA-COPE already addresses pseudo-label ambiguity through geometry but leaves the step size untouched. That makes it the right host: we change only the learning rate and the drift signal, and keep its losses and filtering intact.

REAL275 contains categories with continuous axial symmetry (bottle, bowl, can) alongside asymmetric ones (camera, laptop, mug). This gives a built-in within-benchmark contrast. We will implement $d_G$ per category from the known symmetry annotations.

The work has three parts. The first is the controller above, with $\kappa$ and bounds tuned once on CAMERA-rendered validation streams and then frozen.

The second is a gradual-drift protocol. We build it by slowly ramping photometric and sensor corruptions (illumination, blur, depth noise, color cast) over long REAL275 and T-LESS streams. The drift velocity is known by construction, including piecewise-constant, accelerating, and zero-velocity segments. This addresses a real gap: existing pose adaptation results are reported as a single number on a single shift, and none measure behavior under slow drift. The gradual regime is also exactly where S-TRIGGER and PACE report that threshold-based triggers fail to fire.

The third part is an analysis of why Euclidean proxies fail. We log, frame by frame, parameter displacement, gradient norm, full-$SO(3)$ student–teacher angle, and $d_G$. We then attribute learning-rate spikes to symmetry flips, meaning frames where full-group disagreement is large but $d_G$ is small.

A secondary track tests per-image ambiguity on T-LESS. Here symmetry depends on viewpoint and occlusion, as BOP-Distrib argues, so a fixed $G$ is wrong for some frames. In this setting we replace $d_G$ with a divergence between student and teacher Bingham mixtures (distributions over rotations, used here to represent multi-modal pose uncertainty) in the style of Deep Bingham Networks. The divergence is computed only along directions where the teacher is concentrated. This also sidesteps the tension in the research file between Bingham uncertainty and entropy-style self-training, because the drift signal never asks the model to shrink spread along genuine symmetry axes.

Baselines

The naive reference is the source-only CAMERA-trained network with no adaptation. It shows how much drift actually hurts.

The strongest existing method is TTA-COPE with its published fixed learning rate. On REAL275 it reports 38.1 mAP at the 5°5cm threshold, against 29.1 mAP for TENT (TTA-COPE). We will first reproduce the 38.1 on the standard stream to validate our setup. On our gradual-drift streams there is no reported result found on Lacuna for TTA-COPE or any other method, so all methods will be run by us with identical frames, backbones, adaptation compute, and a matched hyperparameter search. The fixed-rate baseline gets the same number of tuning trials over its single learning rate that our method gets over $\kappa$ and $\eta_0$.

The most important competitor is the Fahrbach et al. controller transplanted into TTA-COPE, using its moving-average parameter-displacement velocity with the same clipping and tuning budget. It is the closest existing implementation of distribution-aware learning rates, and it isolates the value of measuring velocity in quotient function space rather than parameter space.

We add a feature-space drift baseline in the style of Online Change Point Detection on Riemannian Manifolds with Karcher Mean Estimates. That method tracks fast and slow Riemannian means of backbone feature covariances and uses their distance as the velocity signal. It is insensitive to label relabeling because it never looks at outputs.

We also include a reset-on-detection baseline that fires fine-tuning bursts when a KL divergence threshold is crossed, as criticized in PACE.

The key ablation keeps our controller but replaces $d_G$ with the full-$SO(3)$ geodesic angle. This isolates the symmetry quotient itself. A second ablation removes the shuffled noise floor.

Beating the field means three things. The first is higher area under the mAP-versus-time curve than tuned fixed-rate TTA-COPE on gradual streams, across three or more seeds. The second is a statistically clear win over the full-$SO(3)$ ablation on bottle, bowl, and can, with parity on the asymmetric categories. The third is no loss to fixed-rate TTA-COPE on the standard REAL275 stream.

Why not just…?

Why not just tune a fixed learning rate carefully? Fahrbach et al. argue that any fixed rate is a compromise, too slow during fast drift and too noisy during stationary stretches. A single tuned rate can still be hard to beat in practice when drift speed barely varies. Our protocol deliberately includes segments with different velocities and a zero-drift control. If the tuned fixed rate matches us there, the controller adds nothing and we will say so.

Why not just use the Fahrbach displacement proxy as is? On asymmetric objects it may work as well as ours, and we expect parity there. On symmetric objects, the flip mechanism documented by Pitteri et al. predicts parameter movement without physical change. The frame-level attribution analysis and the per-category split will show whether spurious spikes occur and whether they cost accuracy. If they do not, the quotient adds nothing and the paper becomes a negative result about symmetry contamination.

Why not just measure drift on input features, which never see labels? This is the most serious threat, because feature statistics are immune to relabeling by construction. Their weakness is that they respond to any appearance change, including changes the pose network is already robust to, so they may raise the learning rate needlessly and inject pseudo-label noise. We test this with streams that mix task-irrelevant drift (for example, background color) with task-relevant drift (depth noise, blur). If the feature tracker matches us on both kinds of stream, we will adopt it and the quotient output signal is unnecessary.

Why not just detect changes and reset? In gradual drift, step-to-step changes stay below the noise level, and PACE reports that KL thresholds fail to fire. Resets also throw away source knowledge that remains valid under slow drift. The reset baseline on our ramped streams directly measures both failure modes.

Proposed Experiments

Four decisive checks follow from the design.

The first is a zero-drift symmetry stream: REAL275 scenes re-ordered so the camera orbits symmetric objects with no corruption. The ideal learning rate stays at $\eta_0$. We predict the displacement and full-$SO(3)$ controllers will spike on bottles, bowls, and cans, while ours stays flat.

The second uses known-velocity ramps. We report the correlation between each controller's $\eta_t$ trace and the injected drift velocity. Alongside it we report cumulative mAP at 5°5cm and 10°10cm, and the rate of 180° flips per thousand frames.

The third is the per-category split on standard REAL275 against the 38.1 mAP TTA-COPE figure. It checks that the controller does no harm where drift is a single step change.

The fourth is the T-LESS track with Bingham disagreement and visible-surface-discrepancy recall. For context, the offline symmetry-normalized method of Pitteri et al. reports 41.27% VSD recall on T-LESS, against 14.67% for SSD-6D. That is an offline figure, not an adaptation result, so it frames scale only. The check asks whether viewpoint-dependent ambiguity breaks the fixed-group assumption, and whether the distributional signal repairs it.

The proposal fails if the full-$SO(3)$ ablation matches the quotient version on symmetric categories, or if the feature-space tracker matches it everywhere.