A robot looking at a mug with its handle hidden, or at a nearly symmetric part, cannot tell from one image which way the object faces. Several valid orientations look identical. Pose estimators trained only in simulation under domain randomization have two known ways of coping. The first is to average predictions from several robot-commanded viewpoints, which cancels visual noise and beats the roughly 1.5 cm precision limit of single-image estimation. Domain randomization here means training on renderings with random textures, lighting and clutter so the model transfers to real cameras without real labels. The second is to use generative pose models that output a full distribution over orientations, which handles symmetry. These two remedies have never been combined. Averaging destroys the distribution, and the generative models only ever see one frame. We propose to keep a full probability distribution over the object's pose, move the camera, and multiply in each new view's distribution after correcting for the known camera motion. Views that reveal the handle then remove the wrong modes rather than averaging them into a nonsensical orientation. We also propose choosing the next viewpoint to maximally shrink this distribution.
The part that needs research is how much to trust each view. Domain-randomized models are wrong in the real world in correlated ways: the same lighting, the same unfamiliar material and the same sensor dropout affect every view of one episode. A naive product of per-view distributions then counts the same mistake several times and can confidently eliminate the true pose. Our central technical contribution is a tempered product whose tempering is learned entirely in simulation. We generate multi-view episodes in which the randomized nuisance factors are partly shared across views, so the fusion rule learns the degree of cross-view error correlation it will face after zero-shot transfer.
Core Claim
Let $g \in SE(3)$ be the object pose in the robot base frame. Let $T_i$ be the known camera pose for view $i$ from arm kinematics, and let $p_\theta(\cdot \mid x_i)$ be a single-view generative pose model evaluated on image $x_i$ in camera coordinates. We fuse as
$$ \log p(g \mid x_{1:n}) = \beta(n, u) \sum_{i=1}^{n} \log p_\theta(T_i^{-1} g \mid x_i) + \text{const}, $$
where $\beta \in (0,1]$ is a tempering exponent. It depends on the number of views $n$ and on simple episode statistics $u$, such as spread between views and per-view entropy. We fit $\beta$ in simulation by maximizing the held-out likelihood of the true pose over episodes with shared randomization. When $\beta = 1$, the rule is an exact Bayesian product under independent errors. When $\beta = 1/n$, it is a geometric mean that adds no confidence as views accumulate.
We claim three things. First, product-style fusion removes symmetric modes that only some views rule out, which neither averaging nor sample pooling can do. Second, a learned $\beta < 1$ is necessary for fused distributions to stay calibrated on real data. Third, choosing views by expected entropy reduction of this fused distribution reaches a given accuracy with fewer views than the fixed viewpoint schedule used by ITDR.
For objects with true symmetry, the fused distribution should stay multimodal, and that is the correct answer. The payoff is concentrated on near-symmetric objects and on objects whose disambiguating features are self-occluded.
Research Program
Single-view distribution model. We need a single-view model that gives an evaluable density on the pose manifold. RFMPose supplies this method. It learns a flow on $SE(3)$ with geodesic rotation paths, handles symmetric labels with Riemannian optimal transport, and computes exact sample log-likelihoods with the Hutchinson trace estimator. It is trained only on synthetic data and transfers to real data. We reuse it largely as-is and train it on the synthetic SOPE split of Omni6DPose. That split already includes physics-based depth-sensor noise for specular and transparent objects, which serves as our sensor randomization.
Fusion by importance reweighting. Fusion needs no retraining. We draw samples from the most confident view's flow and map them into the base frame with $T_i$. We then reweight them by the tempered sum of the other views' log-likelihoods, and resample. This reuses RFMPose's likelihood machinery in a setting it was never evaluated in, since RFMPose itself is single-frame.
Learning the tempering in simulation. We render multi-view episodes around each object. Within an episode, each randomized factor (texture, lighting, background, depth-noise seed) is shared across views with probability $\rho$ and resampled otherwise, with $\rho$ drawn per episode. We then fit $\beta(n,u)$ by maximizing the log-likelihood of the true pose on held-out episodes.
The key hypothesis is that simulation with sampled $\rho$ yields a $\beta$ that stays calibrated on real episodes. This is analogous to how Neural Posterior Domain Randomization argues that correlated structure in the randomized parameters, not just independent uniform ranges, is what matters for transfer. We borrow the lesson, not the method.
Choosing the next view. Next views are chosen greedily by
$$ v^\star = \arg\max_{v} \; \big( H[p(g \mid x_{1:n})] - \mathbb{E}_{x_v}\, H[p(g \mid x_{1:n}, x_v)] \big), $$
where $H$ is differential entropy estimated from the weighted samples. The expectation over the not-yet-seen image $x_v$ is approximated by rendering the current weighted pose hypotheses from candidate viewpoint $v$ and scoring them with the single-view model. The candidates are a small reachable set on a hemisphere around the object. This keeps planning cheap relative to the flow's integration cost, which matters because GenPose++ tracking already runs at only 17.8 FPS according to Omni6DPose.
Motivating and constraining prior work. Domain Randomization for Active Pose Estimation motivates the whole setting. It shows that moving an eye-in-hand camera, undoing the known motion and averaging deterministic predictions cuts error from 1.57 cm to 0.62 cm with four images. It also cuts in-hand rotation error from 0.129 to 0.047 rad. However, it averages point estimates on simple rigid parts. An Analysis of SVD for Deep Rotation Estimation constrains the design: averaging distinct valid rotations produces invalid or wrong rotations, which is exactly what ITDR-style fusion would do on symmetric objects. Leveraging SE(3) Equivariance for Self-supervised Category-Level Object Pose Estimation reports persistent 180° flips on near-symmetric categories. This is the failure we expect fusion to repair once a disambiguating view arrives.
DiffusionNOCS and GenPose++ represent symmetry-induced multimodality from synthetic training but select a single hypothesis per frame. DiffusionNOCS selects by registration inliers and GenPose++ by DBSCAN clustering. Neither carries a belief across views. We keep the full weighted distribution until the robot must act.
Baselines
Passive single view. The naive reference is RFMPose applied to the first view alone, with no motion. On real ROPE, RFMPose reports IoU25 of 42.1 without real training images, versus GenPose++'s AUC@IoU25 of 39.0 in Omni6DPose. We re-run both on our own objects so that the single-view number is measured, not assumed. Fused results must improve on this with the same backbone, or motion has bought nothing.
Deterministic active averaging. This is ITDR from Domain Randomization for Active Pose Estimation, reimplemented with a deterministic regressor on the same synthetic data and randomization. It uses the same number of views and the same fixed viewpoint schedule as the original. It is the strongest existing active method. Its reported figures (0.62 cm translation with four views; 0.047 rad in-hand rotation) come from simple rigid parts. On symmetric or near-symmetric objects, no reported result was found on Lacuna, so we measure it ourselves. We will also run our method on ITDR-like asymmetric parts to confirm we do not lose precision where averaging is already appropriate.
Pooled generative samples. Per-view samples from RFMPose or GenPose++ are mapped into the base frame, concatenated, and clustered with DBSCAN as in GenPose++. We consider this the most dangerous competitor, because it uses the same distribution-valued model and differs only in the fusion rule. It gets the same backbone, the same views and the same sample budget as our method.
Ablations. Three ablations isolate the key ingredients. The first uses an untempered product ($\beta = 1$). The second uses a fixed geometric mean ($\beta = 1/n$). The third keeps learned tempering but uses random rather than entropy-chosen views. For all three, "beating" means strictly lower symmetry-aware rotation error at matched view count, better calibration (negative log-likelihood of the true pose on real episodes), and fewer views to reach the ITDR precision level.
Why not just…?
Why not just pool samples from each view and cluster them? Pooling takes the union of what each view considers plausible. If view one supports both a handle-left and a handle-right mode and view two sees the handle on the left, the pooled set still contains the handle-right cluster, now with somewhat fewer members. Clustering then has to guess, and this is exactly the heuristic that Omni6DPose's selection step inherits. A product takes the intersection and drives the wrong mode's weight toward zero. This is a real alternative, not a strawman, and on truly symmetric objects the two should tie. Our experiments report results separately for true-symmetric, near-symmetric and self-occluded-feature objects. If pooling matches fusion on the latter two, the central claim fails.
Why not canonicalize each prediction to a symmetric representative and then average, as ITDR does? This works when the symmetry group is known and exact. We include it as a variant of the ITDR baseline with ground-truth symmetry labels. It fails in the interesting cases. For near-symmetric objects, canonicalizing discards the one cue that distinguishes front from back. For self-occlusion, as with a mug's handle, the ambiguity depends on the viewpoint and is not a fixed symmetry group, so there is no representative to canonicalize to.
Why not train one network that takes all views and their relative poses as input? This could work, and it is a threat. It removes the independence question entirely by learning cross-view correlation directly. We train such a multi-view conditional flow on the same synthetic episodes as an additional baseline. We expect it to be tied to the view counts and trajectory distributions seen in training, and it cannot score candidate views without re-running on imagined inputs. If it nonetheless matches our method on accuracy with real, off-distribution trajectories, the modular fusion rule's advantage reduces to view planning, and we will report that honestly.
Why not simply use more views on a fixed schedule? More views help averaging only where errors are independent. The learned-$\beta$ experiment directly measures how much independent evidence real views actually provide. The random-view ablation tests whether chosen views, which specifically target symmetry-breaking features, beat more views.
Proposed Experiments
Real multi-view setup. We replicate ITDR's eye-in-hand and in-hand protocols on a physical arm. The object set spans three groups: ITDR-style asymmetric parts; truly symmetric objects (bowls, cans, bottles from the NOCS categories); and near-symmetric or self-occluding objects (mugs with the handle initially hidden, objects prone to 180° flips). Ground truth comes from fiducials placed outside the camera crop. We report symmetry-aware rotation error, translation error, negative log-likelihood of the true pose, and views needed to reach sub-5 mm and sub-0.05 rad accuracy.
Offline multi-view evaluation. Where camera trajectories are recoverable, we use ROPE tracking sequences from Omni6DPose for passive multi-view fusion, reporting IoU25 and the 5°2cm metric. We also use NOCS Real 275 single-frame results as a check that the backbone reproduces its reported transfer quality. DiffusionNOCS reports 35.0 mAP (5°5cm) on NOCS Real 275 with synthetic-only training.
Decisive checks. Three results would settle the proposal. First, on near-symmetric and occluded-feature objects, product fusion must beat both pooling and ITDR averaging at equal view count; otherwise the core claim fails. Second, on real episodes, learned $\beta$ must be better calibrated than $\beta = 1$; if the untempered product is already calibrated, the correlation-randomization contribution is unnecessary. Third, entropy-chosen views must reach the target precision in fewer moves than ITDR's fixed schedule.
References
- RFMPose: Generative Category-level Object Pose Estimation via Riemannian Flow Matching
- Omni6DPose: A Benchmark and Model for Universal 6D Object Pose Estimation and Tracking
- Neural Posterior Domain Randomization
- Domain Randomization for Active Pose Estimation.
- An Analysis of SVD for Deep Rotation Estimation
- Leveraging SE(3) Equivariance for Self-supervised Category-Level Object Pose Estimation from Point Clouds
- DiffusionNOCS: Managing Symmetry and Uncertainty in Sim2Real Multi-Modal Category-level Pose Estimation