Learning Where to Decouple: Input-Dependent Pivots for Rotation–Translation Uncertainty

Systems that estimate an object's or camera's pose usually split the problem into two parts: where it is (translation) and which way it faces (rotation). Many networks give each part its own branch and its own loss. This split rests on a hidden assumption, which is that errors in the two parts are independent. That assumption depends on an arbitrary choice: the point the rotation is measured about. If the rotation is measured about a point far from the object, a small rotation error drags the translation along with it, like a door swinging on a distant hinge. The two errors then become strongly correlated, and estimating them separately becomes statistically wrong. The best point to rotate about is not fixed. It moves with the input. For a half-occluded object it sits near the visible part. For a camera facing a distant wall it sits out in the scene, not at the camera.

We propose to have the network predict this point, which we call the pivot, for each input. Alongside it, the network predicts rotation and translation with separate input-dependent ("heteroscedastic") error bars. The key observation is that the likelihood used to train separate error bars already tells us where to put the pivot. When rotation and translation are given independent Gaussian errors, the maximum-likelihood pivot is the one that makes the two errors as uncorrelated as possible. Decoupling then becomes a learned, per-sample, testable modeling choice instead of a fixed architectural decision. When the leftover correlation at the best pivot is still large, the model can report that this input should not be decoupled.

Core Claim

Rotation–translation decoupling is justified only about a particular point, and that point depends on the input. Suppose a network outputs a rotation error $\delta\theta$ (a 3-vector in the rotation tangent space) and a translation error $\delta t$, both referenced to the coordinate origin, with joint covariance blocks $\Sigma_{\theta\theta}$, $\Sigma_{t\theta}$, $\Sigma_{tt}$. Re-expressing translation as the displacement of a pivot $p$ gives a new translation error $\delta u = \delta t - [p]_\times \delta\theta$, where $[p]_\times$ is the cross-product matrix of $p$. Its covariances are

$$\mathrm{Cov}(\delta u,\delta\theta) = \Sigma_{t\theta} - [p]_\times \Sigma_{\theta\theta}, \qquad \det \mathrm{Cov}(\delta u) \;\ge\; \det\!\big(\Sigma_{tt} - \Sigma_{t\theta}\Sigma_{\theta\theta}^{-1}\Sigma_{\theta t}\big).$$

The right-hand side is the translation covariance conditioned on rotation, and it does not depend on $p$. Equality holds exactly when the cross-covariance vanishes. A model with separate rotation and translation error bars, trained with Gaussian negative log-likelihood, pays a penalty of $\tfrac12\log\det\mathrm{Cov}(\delta u)$. Minimizing that penalty over $p$ therefore drives the pivot toward the point that decorrelates the two errors. The cross-covariance has nine entries and $p$ has only three, so exact decorrelation is generally impossible. The gap that remains equals the mutual information between rotation and translation errors at the best pivot. That gap is a direct, per-input measure of how wrong decoupling is.

The new element is a pivot $p(x)$ predicted from each input $x$ and trained only through the separate-error-bar likelihood, together with the leftover-correlation diagnostic. Neither appears in the existing decoupling or uncertainty work we reviewed.

Relation to Prior Work

Three decoupling methods supply the architectures we build on, and all of them fix the decoupling point in advance. PoseCNN uses separate heads after a shared backbone, with translation obtained by voting for the object center. FINet uses parallel rotation and translation branches and predicts "saliency points" as rotation centers. That is the closest existing idea to a learned pivot, but those points are trained for registration accuracy and are not tied to any error model. DeTarNet solves translation first, re-centers the points, and then solves rotation. These methods motivate our work, because they show decoupling helps. They do not say where to decouple, or when decoupling is unsafe.

Two papers show that the choice of rotation center matters, and both choose it by hand. Parallel Inversion of NeRFs replaces coupled rigid-motion updates with separate rotation and translation groups to avoid curved "swinging" paths. ReAgent rotates about the point-cloud centroid. Our derivation explains why the centroid works when an object is fully observed, and predicts that it should fail under partial observation.

The uncertainty side has three relevant lines. Geometric Loss Functions for Camera Pose Regression learns one global uncertainty per task to balance rotation and translation, so the same weight applies to every image. HHP-Net predicts per-input variances, but only for rotation, with diagonal error bars, and so never faces the coupling problem. ORORA combines sequential decoupling with directional radar noise models, but its noise models are hand-built from sensor geometry, and nothing feeds back between its stages.

Two results constrain the design. Kendall and Cipolla found that jointly trained position and orientation beat separate models. The calibration study Simultaneous robot-world and hand-eye calibration documents error propagation in two-stage schemes. Our framework takes both results seriously. Both are symptoms of decoupling about the wrong point, and the leftover-correlation diagnostic flags inputs where no pivot rescues decoupling.

The method borrows directly from two more sources. We use the staged training schedule of Learning Local Error Bars: fit the mean first, then the variance, then both jointly. This guards against the variance head inflating to hide poor fits. The noisy-input analysis of Gaussian Process Training with Input Noise shows that input noise passed through a steep function yields input-dependent output noise. Here the lever arm $[p]_\times$ is exactly such a steep, input-dependent slope.

Research Program

We will build a pose head with three outputs.

  1. A pivot. The network predicts $p(x)$ as a 3D offset in the observed frame. It is initialized at the centroid of the input points, or at a fixed depth along the optical axis for camera relocalization.
  2. A rotation. It comes with a full 3×3 input-dependent covariance in the tangent space.
  3. A pivot displacement. It also comes with a full 3×3 input-dependent covariance.

Training minimizes the sum of the two block negative log-likelihoods. The pivot receives gradient only through the translation term, which is what the analysis above requires.

As a companion, we build a "coupled oracle" head that predicts a full 6×6 covariance referenced to the origin. From that covariance we compute the best pivot in closed form, by minimizing the determinant of the pivot-translation covariance over $p$. This gives a reference point for what the learned pivot should approach. We report the leftover mutual information as the decoupling diagnostic.

The analysis has two parts. On synthetic data where the true pivot is known, we check that the learned pivot recovers it. On real data, we test whether the pivot moves the way the theory predicts:

  • toward the visible region as occlusion grows;
  • deeper into the scene for cameras looking at distant structure.

Finally, we plug the learned pivot and covariances into sequential pipelines, both translation-first (DeTarNet-style) and rotation-first (ORORA-style). The test is whether separating at the pivot reduces the error that carries from stage one into stage two.

Baselines

The naive reference is a two-branch network with errors referenced to the origin and a hand-tuned weight between the rotation and translation losses. Its weight gets the same search budget as every other method's hyperparameters. It shows what decoupling buys with no uncertainty modeling.

The first uncertainty baseline is Kendall and Cipolla's learned homoscedastic weighting. It is the standard principled answer to loss balancing. The file reports 90.3% of Cambridge Landmarks queries localized within 2 m and 5°, up from 65.0% for the original PoseNet, and median 7-Scenes position errors between 0.2 m and 0.4 m. Matching its accuracy while adding calibrated per-sample error bars is the minimum bar. Beating it means improving median errors on scenes with large depth variation.

The "obvious idea" baseline extends HHP-Net-style per-input variances to all six degrees of freedom. It keeps the origin as the reference point and uses diagonal error bars. This is the direct ablation of our key ingredient. It has heteroscedasticity but no pivot.

The strongest threat is the same model with a fixed pivot at the centroid of the observed points, as in ReAgent or DeTarNet's re-centering. We include it because it may capture most of the benefit.

The coupled 6×6 heteroscedastic head is the other strong competitor, since it holds the same information without any decoupling.

For registration we compare against the published task methods:

  • FINet: 0.383° rotation error and 0.0066 translation error on unseen ModelNet40 shapes.
  • DeTarNet: mean translation error of 5.317 cm on SUN3D, against 13.13 cm for 3DRegNet.

Our heads are grafted onto each of these backbones without other changes.

All variants share the same backbone, data splits, augmentation, epoch count, and an equal number of hyperparameter trials. All uncertainty variants use the same three-phase schedule. Beating a baseline means four things: lower negative log-likelihood, ellipsoid coverage closer to nominal at 50/90/95%, equal or better point error, and, specifically, gains that concentrate in the low-overlap and high-occlusion bins.

Why not just…?

Why not just rotate about the centroid of the observed points? This is a real threat. For fully observed, roughly uniform objects, the decorrelating pivot should sit near the centroid, and a learned pivot will then add nothing. The theory predicts the two diverge when observation is one-sided: partial overlap, occlusion, or depth-dominated camera views. Our experiments stratify by overlap ratio. If learned and centroid pivots tie in every bin, including the lowest-overlap ones, the method reduces to a justification of the centroid heuristic. We would report it that way, with the diagnostic as the remaining contribution.

Why not just predict a full 6×6 coupled covariance and skip decoupling? That model carries the same information, and the coupled oracle head tests exactly this. The pivot matters when the downstream consumer is decoupled. Sequential solvers, separate refinement steps as in FoundationPose, and robust per-component estimators like ORORA's all need independent rotation and translation uncertainties. The pivot is the reparameterization that supplies them. The error-propagation experiment shows whether this matters in practice. If feeding the full 6×6 covariance into a joint solver beats pivoted decoupling everywhere, decoupling itself is the wrong choice, and we will say so.

Why not just learn per-task weights or tune the balance weight? Kendall and Cipolla themselves found the best weight to be scene-dependent. One global number cannot follow per-image changes such as occlusion, depth, or blur. Even per-image weights leave the swinging cross-term untouched, because rescaling losses does not move the reference point. The diagonal heteroscedastic ablation measures how much of our gain comes from per-sample weighting alone.

Why not just use ensembles or test-time dropout? These capture model uncertainty rather than measurement noise. They also need multiple forward passes, while single-pass error-bar heads run at around 100 FPS (HHP-Net). Robust Neural Regression via Uncertainty Learning reports an average percentage error of 0.066 versus 0.081 for dropout, but on a finance task, so it is weak evidence here. We run a five-member ensemble on the relocalization task as an upper reference for calibration.

Proposed Experiments

Synthetic check. We take ModelNet40 partial-to-partial registration and add programmed direction-dependent noise, along with crops that shift the visible mass. This makes the true decorrelating pivot computable from the injected noise. Success means three things:

  • the learned pivot's distance to the true pivot shrinks with data;
  • the leftover mutual information tracks the injected non-skew cross-correlation;
  • the origin-referenced heteroscedastic baseline shows inflated translation variance proportional to squared pivot distance, as the lever-arm analysis predicts.

Registration. On ModelNet40, SUN3D, and transfer to 7-Scenes, we graft our heads onto FINet and DeTarNet. We report rotation and translation errors against the numbers above, plus negative log-likelihood and coverage, all stratified by overlap.

Camera relocalization. On Cambridge Landmarks and 7-Scenes, we compare against the homoscedastic baseline, inspect where the predicted pivot lands, and measure calibration.

Decisive tests.

  1. Error propagation. In translation-first and rotation-first sequential pipelines, we regress stage-two error on stage-one error. A clearly smaller slope when separating at the learned pivot, compared with the centroid and the origin, supports the core claim. No change refutes it.
  2. Swinging. In iterative refinement, we compare refinement path length and the number of non-monotone translation steps.
  3. Abstaining on high-correlation inputs. We check whether the diagnostic predicts failure: inputs with high leftover correlation should be exactly those where the coupled head beats the decoupled one.

If that last correlation is absent, the diagnostic is useless, and the contribution shrinks to a better parameterization.