When a robot looks at a cup without a handle, the image does not determine how the cup is rotated about its vertical axis. Pose estimators trained to output a single rotation handle this badly: they average the equally valid answers and produce a rotation that matches none of them. Pose-estimation research calls this mean collapse. The standard fixes need a CAD model (an exact 3D mesh of the object) to recognize which rotations are equivalent, and a CAD model is exactly what is missing for an object named in free-form text. We propose that the information bottleneck can solve this problem instead of causing it. The information bottleneck is a training penalty that forces a representation to keep as little information about the input as the task allows. The usual objection is that such a penalty accelerates mean collapse. That is true only when the representation is a Gaussian point in Euclidean space. If the representation is a probability distribution over rotations, and the task signal only asks that sampled rotations align the object's observed depth points across two views, the cheapest representation is the one that spreads probability uniformly over every rotation the data cannot distinguish. Compression then becomes a mechanism for finding, per image, the set of ambiguous rotations. It needs no CAD mesh, no symmetry labels, no category list, and no pixel reconstruction.
In two sentences: a text-selected object's relative pose between two RGB-D views is predicted as a distribution on the rotation group. It is trained only to make sampled rotations align the observed depth points, plus a penalty that pushes the distribution toward uniform. The alignment term rules out wrong rotations, the penalty spreads mass over all rotations that remain indistinguishable, and the result is a single-forward-pass estimate of which rotations are equivalent for objects the model has never seen.
Core Claim
Let $x = (I_A, I_B, \text{text})$ be an anchor and a query RGB-D image with a text prompt naming the object, and let $P_A, P_B$ be the object's observed depth points in each view. The network outputs a Bingham distribution $q_\theta(R \mid x)$ over the relative rotation $R$. A Bingham distribution is a Gaussian-like density on unit quaternions that can be sharp along some rotation axes and flat along others; Gilitschenski et al. introduced it as a deep learning loss. Translation $t$ is regressed separately, since it is not affected by rotational symmetry. We train the constrained objective
$$\max_\theta \; H\big(q_\theta(R \mid x)\big) \quad \text{s.t.} \quad \mathbb{E}_{R \sim q_\theta}\, d_{\text{CD}}\big(R P_A + t,\; P_B\big) \le \varepsilon,$$
where $H$ is entropy, $d_{\text{CD}}$ is a truncated two-sided Chamfer distance between the partial point clouds restricted to their overlap, and $\varepsilon$ is set to the measured depth-sensor noise floor. This is the information bottleneck with a uniform prior on rotations. The KL divergence to a uniform prior equals a constant minus entropy, so $\beta$ in $\max I(Z;Y) - \beta I(X;Z)$ from Tishby et al. becomes the Lagrange multiplier of this constraint. It is fitted by dual ascent rather than hand-tuned.
The mechanism is easy to see. Without the entropy term, any distribution concentrated on the zero-loss set minimizes the alignment loss, so the network collapses to one arbitrary mode. This is not an average, because we sample rather than regress, but it hides the ambiguity. With the entropy term, the optimum is the maximum-entropy distribution on the set of rotations that satisfy the alignment constraint. That set is the per-view visual ambiguity set: a full circle for a smooth bowl, discrete modes for a box, a single point once a handle is visible.
The alignment term plays the role of PoseCNN's ShapeMatch-Loss, which is zero for every symmetry-equivalent rotation. The difference is that our term uses the sensor's observed partial geometry instead of a mesh. Because only visible points enter the loss, a self-occluded asymmetric object also comes out as ambiguous. This is the viewpoint-dependent ambiguity that BOP-Distrib showed global CAD symmetries miss.
Positioning
Several papers motivate this proposal by exposing the gap. Mahendran et al. and Pitteri et al. show that regression losses average symmetric solutions into invalid rotations. BOP-Distrib shows that true ambiguity is a per-image distribution, but it computes that distribution from CAD models, and only for evaluation. The open-vocabulary pose estimators do not model ambiguity as a distribution; each handles it partially or not at all. Oryon matches CLIP and Swin features and registers them with PointDSC, which splits into competing clusters on symmetric objects. ConceptPose breaks symmetry with LLM-named parts found by Grad-CAM. It fails on featureless symmetric shapes and takes about 7 seconds per pair. Cai et al. regress dense normalized-coordinate maps using symmetry-axis annotations, which is a label we refuse to require.
Other work constrains the design. Information bottleneck methods in control, most directly DRIBO, use Gaussian latents and a global compression term. The research file notes that DRIBO discards small localized features, which is why we apply compression only to the rotation distribution and never to appearance features. The reconstruction literature also shapes the design. Shelhamer et al., Du et al. and Schwarzer et al. argue that reconstruction objectives spend model capacity on irrelevant detail. The Mask World Model shows that predicting geometry rather than RGB helps manipulation, while noting that silhouettes carry no orientation information for symmetric objects. We therefore predict no pixels, no masks and no coordinate maps. The only target is observed geometry, and it enters only through the alignment check.
We reuse two methods directly. The Bingham parameterization comes from Gilitschenski et al.. Text-conditioned object selection comes from Oryon's CLIP cost volume, which we keep frozen.
The closest single prior idea is kPAM. kPAM shows that manipulation only needs the symmetry axis of a cup or bowl, not its full rotation, but it learns axis keypoints per category with supervision. Our flat Bingham directions produce that axis for unseen categories without labels.
Research Program
The encoder reuses Oryon's frozen CLIP text and image cost volume, followed by a light transformer over back-projected features from both views. It outputs Bingham parameters: a 4×4 orthogonal frame and three concentrations. Translation is regressed from the matched feature centroids.
Training uses ShapeNet6D anchor–query pairs rendered with depth, the same source Oryon trains on. Ground-truth poses are used only in ablations. Sampling from the Bingham uses reparameterized rejection sampling, with 32 samples per pair. The Chamfer term uses a truncation radius tied to $\varepsilon$ so that non-overlapping regions do not dominate the loss.
We will analyze three questions. First, whether the learned flat directions match BOP-Distrib ambiguity sets computed from CAD models the network never sees. Second, how the learned multiplier $\beta$ varies across objects, since it should be large on asymmetric shapes and irrelevant on symmetric ones. Third, what happens when $\varepsilon$ is set below the true noise level. The prediction is that the distribution collapses to a delta. Setting $\varepsilon$ too high should wash out real modes, which is the fine-feature risk flagged for DRIBO.
Baselines
The naive reference is the same network trained with a geodesic rotation loss against the single annotated ground-truth pose. This is the regime where Pitteri et al. predict mean collapse. It shows what the problem costs on symmetric objects. We beat it if symmetric-subset error drops while error on asymmetric objects does not rise.
The strongest existing open-vocabulary system on accuracy is ConceptPose. It reports 71.5% ADD(-S) on REAL275, 55.0% on Toyota-Light and 41.2% on YCB-Video, at about 7 seconds per pair. We run it with its released SigLIP2 backbone on identical anchor–query pairs. Our claim is not higher overall ADD(-S), which is itself a CAD-dependent, symmetry-tolerant metric. It counts as beating ConceptPose if we match it within a few points while running at least two orders of magnitude faster, and clearly exceed it on the smooth symmetric subset where its concept maps become uniform.
The strongest learned open-vocabulary competitor with the same backbone is Oryon. It reports an average recall of 32.2 on REAL275 and 30.3 on Toyota-Light. Because we share its frozen features and training data, this comparison isolates the pose head.
The ablations that isolate the key ingredient share the architecture, data and tuning budget. The first sets $\beta = 0$, keeping Chamfer alignment without entropy. It should be accurate on its chosen mode but uninformative about ambiguity. The second is a Gaussian-latent variational bottleneck with the same Chamfer term, which tests the claim that a Euclidean prior drives collapse. The third is a supervised Bingham fit by maximum likelihood on ground-truth poses.
For distribution quality we found no reported result on Lacuna for LiePose-Diffusion or SpyroPose under BOP-Distrib on T-LESS. We will run both as CAD-trained upper references.
Why not just…?
Why not just fit a Bingham or mixture by maximum likelihood on ground-truth poses? This is the real threat. Given enough images, maximum likelihood can learn the spread of annotated poses in aggregate. It needs pose labels, however, and for novel objects it has to infer an instance's symmetry from dataset-level annotation habits. With few views per shape, it can simply memorize the arbitrary pose each annotator chose. We run exactly this model on the same ShapeNet6D pairs with ground-truth poses. If it matches our ambiguity-set recovery on held-out categories, the contribution reduces to removing the need for labels, and we will say so.
Why not just reconstruct a mesh from the anchor views and apply ShapeMatch? This removes the CAD requirement, but it gives only an arbitrary zero-loss mode unless something forces the result to spread. It also adds multi-view reconstruction at test time. It is equivalent to our $\beta = 0$ ablation plus a reconstruction step, so the entropy term's contribution is measured directly.
Why not just use ConceptPose and treat failures as noise? Its reported latency rules it out for closed-loop control. Its failure mode also concentrates on exactly the smooth symmetric objects we target. The symmetric-subset comparison will show whether that gap is real.
Why not just compress with a standard Gaussian variational bottleneck? The research file's Gap 2 argues that pulling the latent toward a unimodal Euclidean prior worsens averaging. The Gaussian ablation tests this directly. If it does not collapse, our argument for the rotation-space prior fails.
Proposed Experiments
The decisive check uses BOP-Distrib's per-image ground-truth distributions on T-LESS and YCB-Video, which are available for evaluation only, trained on ShapeNet6D, with no T-LESS training data. We measure recall under the BOP-Distrib protocol and the coverage of the ambiguity set, meaning the fraction of equivalent rotations that receive probability mass. The prediction is that the $\beta = 0$ and geodesic baselines cover a single mode, the Gaussian bottleneck produces invalid averages, and our model's coverage tracks the ground-truth set, including views where a handle is hidden.
Second, we evaluate anchor–query pose on REAL275 and Toyota-Light with average recall and ADD(-S), reporting separately on objects with and without rotational symmetry and recording wall-clock latency. Third, on OO3D-9D we test whether the flat Bingham directions recover the annotated symmetry axes of novel categories. That dataset reports 87.8% IoU@50 mAP for the Cai et al. system on a different task, so we use only its axis annotations as held-out labels.
Finally, we run a small downstream test in RLBench place-and-insert tasks involving cups and pegs. A kPAM-style planner receives either our axis or a single point estimate of the rotation. For symmetric-object placement in this setting we found no reported result on Lacuna, so both arms are run by us. If axis-aware planning does not improve success over the point estimate, the manipulation motivation is weaker than claimed. The pose results would still stand.
References
- Deep Orientation Uncertainty Learning based on a Bingham Loss
- Data Clustering by Markovian Relaxation and the Information Bottleneck Method
- PoseCNN: A Convolutional Neural Network for 6D Object Pose Estimation in Cluttered Scenes
- BOP-Distrib: Revisiting 6D Pose Estimation Benchmarks for Better Evaluation under Visual Ambiguities
- 3D Pose Regression Using Convolutional Neural Networks
- On Object Symmetries and 6D Pose Estimation from Images
- Open-vocabulary object 6D pose estimation
- ConceptPose: Training-Free Zero-Shot Object Pose Estimation using Concept Vectors
- Open-Vocabulary Category-Level Object Pose and Size Estimation
- DRIBO: Robust Deep Reinforcement Learning via Multi-View Information Bottleneck
- Loss is its own Reward: Self-Supervision for Reinforcement Learning
- Denoised MDPs: Learning World Models Better Than the World Itself
- Predicting What Matters: Robust Generalist Robot Policy Learning via Future Semantic Mask
- KPAM: KeyPoint Affordances for Category-Level Robotic Manipulation