Robots that learn to recognize objects in simulation fail in quiet ways once they see real sensor data. A pose estimator trained on rendered CAD models (computer models of each object's 3D shape) can return a confident answer for a real depth image, and that answer can be wrong. Sometimes the observation genuinely cannot pin down the pose: a mug with its handle hidden looks the same from many angles. Sometimes the real object simply does not match the model the simulator assumed. This note proposes a checker that, for each real observation, either issues a short proof that the estimated pose is within a tolerance $\zeta$ of the truth, or issues a short proof of why no such guarantee is possible.
The idea comes from certifying algorithms, a tradition in algorithm design where a complex, untrusted program returns its answer together with a witness: a piece of evidence that a small, separately trusted program can check quickly. The classical rule in that tradition is that both "yes" and "no" answers carry witnesses. We apply this rule to synthetic-to-real transfer. An untrusted search produces one of three checkable objects. The first is a uniqueness witness: a certified lower bound showing every pose farther than $\zeta$ from the estimate fits the data badly. The second is an ambiguity witness: an explicit second pose, far from the first, that fits equally well. The third is a mismatch witness: a certified bound showing no pose fits the data within the sensor tolerance at all. This removes the hand-crafted keypoint sets that current certifiable pose estimators depend on. It also turns rejected real samples from silent discards into diagnostic evidence about where the simulator is wrong.
Core Claim and How It Differs From Prior Work
The closest work is C-3PO, presented in Certifiable Object Pose Estimation: Foundations, Learning Models, and Self-Training and Correct and Certify. It trains a keypoint-based pose network on synthetic shapes and defines a pose as $\zeta$-correct when it lies within $\zeta$ of the truth. It proves that two runtime checks together imply $\zeta$-correctness. The first, "observable correctness," asks whether the aligned CAD model matches the observed points. The second, "non-degeneracy," asks whether the detected keypoints constrain all six degrees of freedom. Poses that pass are then used as pseudo-labels to self-train on real YCB point clouds.
This line of work motivates our proposal and supplies its self-training loop. Its own stated limitations define our target. The non-degeneracy check uses hand-crafted indicator sets of keypoints. The tolerance $\epsilon_{oc}$ is tuned by hand. The checker is unverified floating-point code. A rejection produces nothing beyond "reject."
We replace the hand-crafted non-degeneracy check with a computed one that needs no keypoints. Let $r(T)$ be a robust residual between the observed point cloud and the CAD model placed at pose $T$, for example a truncated one-sided Chamfer distance. Make one explicit physical assumption, the sensor envelope: the true pose $T^*$ satisfies $r(T^*) \le \epsilon$, where $\epsilon$ bounds sensor noise and simulator-to-real geometric error. Under that assumption, the following statement carries a guarantee:
$$\big[\, r(\hat T) \le \epsilon \;\text{ and }\; r(T) > \epsilon \;\; \forall T: \mathrm{dist}(T, \hat T) \ge \zeta \,\big] \implies \mathrm{dist}(\hat T, T^*) < \zeta .$$
The second condition is what the uniqueness witness establishes. The witness is a finite cover of the far region of pose space by cells $C_k$, each with a lower bound $L_k = r(c_k) - \lambda\,\rho_k > \epsilon$. Here $c_k$ is the cell center, $\rho_k$ is its radius, and $\lambda$ is a Lipschitz constant of $r$, which is bounded by the object's radius in the standard rotation-search argument. Translation is bounded a priori, because observed points must lie on the model and therefore within one object diameter of their centroid.
The checker only has to do three things: confirm the cells come from a valid subdivision tree that covers the far region, evaluate $r$ at each center, and compare. This is linear in the number of cells times the number of points.
The two "no" outcomes are equally cheap to check. An ambiguity witness is a pose $T'$ with $r(T') \le \epsilon$ and $\mathrm{dist}(T', \hat T) \ge \zeta$, measured modulo the object's known symmetry group. Checking it means running observable correctness once more. A mismatch witness is a cover of the entire bounded pose space with all $L_k > \epsilon$. It proves the sensor-envelope assumption fails for this observation. Typical causes would be a wrong CAD model, deformation, or segmentation bleeding into the background. These are exactly the sim-to-real failures that pseudo-labeling methods silently absorb.
This differs from global solvers for 3D vision, surveyed in Advances in Global Solvers for 3D Vision. Those methods supply our branch-and-bound machinery and certify that an estimate is the global optimum of an objective. They do not certify that the optimum is well separated from other near-optimal poses. Separation, not optimality, is what implies closeness to the truth when the observation is degenerate. The survey also names the gap we fill: certifiable backends are paired with learned front-ends that are exposed to domain shift.
The proposal also differs from Provably Safe Sim-to-Real Transfer. That paper certifies regions where a simulator's dynamics match reality, using statistical confidence bounds in a tabular decision process. We produce instance-level, deterministic witnesses on continuous, high-dimensional observations. Finally, certified out-of-distribution methods such as Certifying Out-of-Domain Generalization for Blackbox Functions and GSmooth bound error under norm-bounded or parameterized perturbations and yield no per-instance witness for an independent checker. Our guarantee depends only on geometry and the stated envelope assumption, not on any model of the shift.
Research Program
The work has four parts.
The first part is witness generation. A GPU branch-and-bound over rotation and bounded translation searches the region outside the $\zeta$-ball around the network's estimate, with the ball defined modulo symmetry. It prunes cells whose lower bound exceeds $\epsilon$, and it stops at the first cell center with $r \le \epsilon$, which becomes an ambiguity witness. If $\hat T$ itself fails observable correctness, it runs a second search over the full space that ends either in a better pose or in a mismatch witness. When the compute budget runs out, the result is reported as "inconclusive" rather than forced into a verdict. We will measure how often this happens. The pruning and filtering engineering follows the lesson of Walking the dog fast in practice: there, certificate generation for Fréchet decisions cost only about 50% over the plain decider once search pruning was engineered carefully.
The second part is a small trusted checker. Following the two-tier method of Verification of Certifying Computations, we will prove the witness property in Isabelle/HOL: cover validity plus all bounds above $\epsilon$, together with the envelope assumption, implies $\zeta$-correctness. We will also verify the checker code. Floating-point issues will be handled with interval arithmetic in the residual evaluation. We treat full code verification as a stretch goal. The mathematical witness-property proof and the soundness fuzzing described under the experiments are the committed deliverables.
The third part is self-training with three outcomes. It extends the C-3PO loop. Uniquely certified poses become pseudo-labels. Ambiguity witnesses become set-valued labels, meaning the network is trained to predict any pose in the ambiguous family rather than a single one. Mismatch witnesses are logged with the specific observed points that violate the bound.
The fourth part is the diagnostic use of those witnesses. Because a mismatch witness localizes the offending points, aggregating witnesses across the real set gives an unlabeled estimate of where the simulator's geometry is wrong. The rate of mismatch witnesses as a function of $\epsilon$ also offers a principled replacement for C-3PO's hand-tuned tolerance. We will test that replacement rather than assume it works.
Baselines
The naive reference point is the network trained only on synthetic data, with its raw outputs used directly and no certification or self-training. This measures the size of the sim-to-real gap on our split.
The strongest existing method is C-3PO with its keypoint corrector, observable-correctness check, and hand-crafted non-degeneracy check, run through the same self-training rounds. Its authors report that on the mug object, applying both certificates raised ADD-S from 68.00% to 100.00% (Correct and Certify). ADD-S is the standard pose-accuracy score: the average distance between points on the model placed at the estimated pose and their nearest points on the model placed at the true pose. We will rerun C-3PO on our exact split and use our own run as the comparison, quoting the published figure only for context.
To keep the comparison fair, C-3PO and our method will share the backbone, the synthetic training data, the residual $r$, the tolerance $\epsilon$, and the number of self-training rounds. Both get the same per-instance wall-clock budget, so C-3PO's spare budget can go to more corrector iterations. Beating C-3PO means a lower false-certification rate at equal or higher certified yield, or equal false certification with substantially higher yield. Here false certification means a pose is certified but has error above $\zeta$, and certified yield means the fraction of real instances that get certified. It also means higher ADD-S after self-training on objects for which C-3PO has no hand-crafted indicator set.
Two further reference points come from supervised pipelines. PoseCNN reports an RGB ADD-S AUC of 75.9 on YCB-Video. CosyPose reports 89.8 with one view, rising to 93.4 with five. These are supervised RGB numbers in a different training regime, so they are context and not a target.
Two ablations isolate the key ingredient. The first, optimality-only, uses the same branch-and-bound to certify only that $\hat T$ is the global minimizer of $r$, without excluding the far region. If this ablation matches our false-certification rate, then separation adds nothing. The second, observable-correctness-only, removes the non-degeneracy component entirely. It shows how much of the gain comes from degeneracy handling at all.
Why not just…?
Why not just use domain randomization so the real world looks like another simulated sample? Randomization narrows the gap but gives no per-instance guarantee. Robust Visual Sim-to-Real Transfer for Robotic Manipulation needed a separate proxy task just to tune its randomization ranges. Grasp Stability Prediction with Sim-to-Real Transfer from Tactile Sensing found accuracy collapses under slight friction miscalibration. Randomization is complementary to our checker, not a substitute for it. We will train the backbone with randomized rendering in all conditions, so any gain we report is on top of it.
Why not just threshold softmax or ensemble confidence, as in Mean Teacher pseudo-labeling? Self-ensembling for visual domain adaptation reached 92.8% on VisDA-2017 with a confidence threshold above 96.8%. That result relied on a class-balance loss. More fundamentally, confidence has no witness property: a confident prediction can be wrong for reasons the threshold cannot see. Our experiments include an ensemble-agreement filter at matched yield. Its false-certification rate on symmetric and occluded instances is exactly what we expect to be high.
Why not just run ICP from many starting poses and declare the answer unique if they all agree? This is the real threat. Empirically it may approximate our uniqueness test well and cost less. Correct and Certify shows plain ICP refinement fails under large sim-to-real keypoint errors, but multi-start agreement is a stronger heuristic than single-start refinement. We will run it with the same compute budget as our search and compare false-certification rates. If the rates match on YCB, the remaining contribution is the guarantee and the mismatch diagnostics, and we will say so. We expect multi-start agreement to diverge from our test on near-symmetric objects with thin distinguishing features, such as a partially visible handle, where few starts land in the narrow basin. Our injected-occlusion test targets exactly that case.
Why not just use a global solver's optimality certificate? As Advances in Global Solvers for 3D Vision describes, a tight relaxation proves the estimate is the global minimum of the objective. On a degenerate observation, though, the global minimum can be one of several nearly tied poses, so optimality does not imply closeness to the truth. The optimality-only ablation tests this directly.
Proposed Experiments
All experiments follow the C-3PO protocol. The network trains on synthetic renderings of CAD models and adapts without labels on real depth from the YCB objects. Ground-truth poses from YCB-Video are used only for evaluation. The metrics are ADD-S and ADD-S AUC, the false-certification rate, certified yield, the ambiguity, mismatch, and inconclusive rates, and witness generation and checking times.
The first decisive check is soundness under stress. Across all real instances, the number of certified poses with error above $\zeta$ should be zero, apart from instances where the ground truth itself violates the envelope assumption. We will report those separately by measuring $r(T^*)$ directly. C-3PO and multi-start ICP should show a nonzero count, concentrated on symmetric objects and on views we artificially occlude by cropping 30–70% of the points.
The second check is the mismatch witness. We will deliberately pair observations with the wrong CAD model, or with a CAD model scaled by 10–20%. The mismatch witness should fire at high rates, while confidence-based filters should continue to emit pseudo-labels.
The third check is whether the method scales to objects without engineering. On YCB objects for which no hand-crafted indicator sets exist, self-training with our witnesses should improve ADD-S over the source-only model. C-3PO cannot be applied to these objects without manual work.
The fourth check is cost. Checking must be at least an order of magnitude faster than generation; that asymmetry is the point of the certifying design. We will also report the generation-time distribution and the inconclusive rate as functions of $\zeta$. This may expose the same scalability limits the global-solver survey reports.
Finally, we will fuzz the checker by corrupting witnesses: dropping cells, shrinking bounds, and perturbing centers. Every corrupted witness must be rejected. The Isabelle/HOL proof of the witness property will be reported along with its size and the time it takes to check.
References
- Certifiable Object Pose Estimation: Foundations, Learning Models, and Self-Training
- Correct and Certify: A New Approach to Self-Supervised 3D-Object Perception
- Advances in Global Solvers for 3D Vision
- Provably Safe Sim-to-Real Transfer
- Certifying Out-of-Domain Generalization for Blackbox Functions
- GSmooth: Certified Robustness against Semantic Transformations via Generalized Randomized Smoothing
- Walking the Dog Fast in Practice: Algorithm Engineering of the Fréchet Distance
- Verification of Certifying Computations
- PoseCNN: A Convolutional Neural Network for 6D Object Pose Estimation in Cluttered Scenes
- CosyPose: Consistent multi-view multi-object 6D pose estimation
- Robust Visual Sim-to-Real Transfer for Robotic Manipulation
- Grasp Stability Prediction with Sim-to-Real Transfer from Tactile Sensing
- Self-ensembling for visual domain adaptation