Negotiated Pseudo-Labels for Sim-to-Real Pose Estimation in Cluttered Bins

A pose estimator trained only on rendered images usually breaks on real camera data, so a standard repair is self-training. The simulator-trained model labels unlabeled real images, the confident labels are kept, and the model is retrained on them. Everything depends on which labels are kept. Current systems accept a predicted object pose only if it passes separate fixed tests on the color image and on the depth point cloud. Each object in the bin is judged alone. In a cluttered bin of shiny metal parts this goes wrong in predictable ways. Reflections corrupt the depth data even when the 2D outline matches well. Neighbouring parts compete for the same pixels. A pose that looks fine in isolation can explain a region that clearly belongs to another object.

We propose to choose pseudo-labels for a whole scene jointly with dual decomposition. Dual decomposition, also called Lagrangian relaxation, splits a hard joint problem into easy pieces and solves each piece separately. It then adjusts per-variable "prices", the Lagrange multipliers, until the pieces agree. In our setting the pieces are a 2D appearance solver per object, a 3D geometry solver per object, and a scene-level solver that assigns each pixel and depth point to at most one object. Agreement then becomes a concrete test. When all pieces agree, the duality gap is zero: the gap between the price-adjusted bound and the score of the agreed solution vanishes. This certifies that the scene labeling is the exact optimum of the joint energy, with no hand-set acceptance threshold. When the pieces do not agree, the leftover prices show which objects, and which pixels or points, are in conflict.

Core Claim

We claim two things. First, scene-level exclusivity coupling, not modality fusion alone, is what recovers correct pseudo-labels in clutter. Second, zero-gap agreement is a better acceptance signal than fixed per-modality thresholds, judged by pseudo-label precision at matched recall. The agreement certificate is a statement about the energy. It does not prove the pose is correct. Whether certified labels are actually more accurate is therefore an empirical question, and it is the central one we test.

Relation to Prior Work

Two sim-to-real papers motivate the problem. Sim-to-Real 6D Object Pose Estimation via Iterative Self-training for Robotic Bin Picking builds the self-training loop we start from: a teacher trained on rendered scenes, and pseudo-labels kept only if they pass both a 2D perceptual mask distance and a 3D Chamfer distance against the depth cloud. Those two tests are fixed thresholds applied to each instance independently. A sample on which they disagree is dropped.

The research file reads that paper's "40% of unlabeled data" result as meaning that 60% of samples get discarded. The paper actually reports data efficiency, so we do not assume a particular rejection rate. We will measure the rejection rate directly.

Certifiable Object Pose Estimation: Foundations, Learning Models, and Self-Training (C-3PO) filters self-training data with an "observable correctness" certificate built on a manually chosen tolerance, and a hand-crafted non-degeneracy check. It also treats each object alone. It is the strongest existing argument that certificates make good filters, and our work builds directly on that idea. Our certificate differs in two ways. It comes from agreement across a whole scene rather than a single-object tolerance. It also needs one scene-independent rejection cost in place of a per-check tolerance.

The method itself comes from the dual decomposition literature. A Tutorial on Dual Decomposition and Lagrangian Relaxation for Inference in Natural Language Processing and On Dual Decomposition and Linear Programming Relaxations for Natural Language Processing supply the subgradient negotiation and the agreement-implies-optimality certificate. Multi-level inference by relaxed dual decomposition for human pose segmentation is the closest structural template. It couples a pixel segmentation solver with a limb-pose solver through a term that penalizes unexplained pixels, and it negotiates over finite candidate pools. That work stays inside a single real domain and does no self-training.

Robust Biomedical Event Extraction with Dual Decomposition and Minimal Domain Adaptation. shows that coupling sub-models through dual decomposition transfers across text domains. There, though, the transfer comes from mixing training data, not from the multipliers. We use the multipliers and the duality gap themselves as the transfer mechanism, deciding which real labels to keep.

One prior result constrains the design. The file notes that with integer variables the relaxation can be loose, leaving a nonzero gap and causing the negotiation to oscillate. We therefore treat "how often is the gap zero on real bins" as a measured quantity, not an assumption.

Research Program

Assume the simulator-trained teacher has proposed $K$ object instances in a real RGB-D scene. For each instance we build a finite pool of candidate poses. The pool contains the teacher's top hypotheses, small perturbations of them, and an explicit "reject" option. The joint labeling problem is

$$\min_{z}\ \sum_{k=1}^{K}\big[E_{2D}(z_k)+E_{3D}(z_k)\big]\quad \text{s.t. each pixel and each depth point is explained by at most one } z_k.$$

The symbols are as follows. $z_k$ is the chosen candidate for instance $k$. $E_{2D}$ is the rendered-mask appearance distance from the bin-picking paper. $E_{3D}$ is the robust Chamfer distance to the observed cloud. A rejected instance pays a constant cost $\lambda$, which is calibrated once on synthetic validation scenes.

We relax the problem with three groups of copies of the decision variables. The 2D solver chooses a candidate per instance using appearance alone. The 3D solver chooses one using geometry alone. A scene solver assigns each pixel and point to a single owner or to background, which can be done independently per pixel and per point. Prices enforce that the 2D and 3D candidate choices agree and that the rendered masks and point sets match the scene solver's assignment. Every subproblem is either an argmin over a small pool or a per-pixel argmin. Each iteration therefore costs roughly as much as running the two existing filters once.

After a fixed iteration budget there are two kinds of acceptance. A scene whose gap has closed is accepted in full. Otherwise, an instance is accepted when its local copies agree and its share of the remaining prices is small.

The prices are also a diagnostic. Large accumulated prices on depth points flag regions where the 3D evidence keeps losing to the 2D evidence. We will check whether these regions line up with reflective surfaces in ROBI. If they do, the pattern would give a per-material reliability map that a later round of self-training could use.

Baselines

The naive reference is the teacher trained only on synthetic data, with no adaptation. Every gain is measured from this point.

The strongest existing method is the two-threshold self-training pipeline of Sim-to-Real 6D Object Pose Estimation via Iterative Self-training for Robotic Bin Picking. It is the system built for exactly this benchmark and this failure mode. That paper reports ADD(-S) improvements over the synthetic-only teacher of 11.49% on ROBI and 22.62% on SIBP. The research file does not give absolute ADD(-S) values, so we will reproduce them. The paper also reports physical grasp success on a Franka Emika Panda rising from 67.96% to 87.50%. We will rerun this pipeline with the same teacher, the same candidate pools, and the same number of self-training rounds. Its thresholds get the same tuning budget on synthetic validation scenes that our $\lambda$ gets. Beating it means higher final student ADD(-S) on the real test split, and higher pseudo-label precision at equal or greater recall.

A C-3PO-style baseline applies the observable-correctness certificate from Certifiable Object Pose Estimation: Foundations, Learning Models, and Self-Training to each instance independently, with its tolerance tuned under the same budget. It represents the certificate-as-filter idea without any scene coupling.

The most dangerous baseline is per-instance weighted fusion. It picks the candidate minimizing $E_{2D}+\beta E_{3D}$ for each instance over the same pool, then thresholds the result, with $\beta$ and the threshold tuned under the same budget. It isolates whether modality fusion alone explains any gain.

A greedy sequential scene fitter claims objects one at a time in order of confidence and removes the pixels each one explains. This is the pipeline alternative that does model exclusivity. If our method beats it, the improvement comes from joint negotiation rather than from the exclusivity constraint alone.

Two ablations isolate the key ingredient. One drops the scene solver and keeps only the 2D–3D agreement; this reduces to a negotiated version of per-instance fusion. The other keeps the full scene coupling but accepts by thresholding final energies instead of using agreement and the gap.

Why not just…?

Why not just tune a weighted sum of the 2D and 3D scores per object? This is the real threat. If clutter is not the main source of bad labels, per-object fusion over the same candidate pool will match our method. In that case dual decomposition would add only machinery. The first ablation settles it directly. We will also stratify results by occlusion level. Our claim predicts the gap between the two opens only in heavily occluded scenes. If it does not open there, we will report that scene coupling is unnecessary.

Why not just fit objects greedily, most confident first? Greedy fitting handles exclusivity, but it inherits the pipeline weakness that the dual decomposition literature documents (A Tutorial on Dual Decomposition and Lagrangian Relaxation for Inference in Natural Language Processing). An early wrong claim on a reflective part takes pixels from its neighbours, and nothing later can give them back. The greedy baseline measures how often that happens on ROBI and SIBP. If greedy fitting matches our method, joint negotiation is not worth its iterations.

Why not just use C-3PO's certificate with a better-tuned tolerance? The C-3PO certificate already filters well for single objects, but it cannot see that two individually plausible poses claim the same depth points. We compare the two certificates by plotting pseudo-label precision against recall over each method's single free parameter. That plot shows whether scene agreement adds information beyond tolerance tuning.

Why not just do adversarial feature alignment? Alignment in feature space, as in Unsupervised Contrastive Domain Adaptation for Semantic Segmentation, reports 60.2% mIoU on GTA5→Cityscapes, but it still struggles on small, hard classes. More importantly, it gives no per-sample signal for which real labels to trust. It is complementary to our approach and could be applied to the teacher. It is outside the main comparison, which is about label selection.

Proposed Experiments

We use SIBP, which has 6,000 synthetic and 2,743 real RGB-D scenes, and ROBI, which contains reflective metal parts. Both come from Sim-to-Real 6D Object Pose Estimation via Iterative Self-training for Robotic Bin Picking. Real ground-truth poses are hidden during self-training and used only for evaluation.

The first decisive check is calibration. On the real training split, we measure what fraction of certified labels lie within the ADD(-S) tolerance, compared with labels that pass the thresholds. We also measure how often the gap closes, broken down by occlusion level and by object material. If certified labels are not more accurate, the core claim fails, whatever the downstream numbers show.

The second check is downstream accuracy. We compare final student ADD(-S) on the real test splits after the same number of rounds, against every baseline and ablation above. We also report iterations and wall-clock time per scene.

The third check tests the diagnostic. On ROBI, we measure whether accumulated depth-point prices concentrate on reflective regions. We will measure this against depth error computed from ground-truth poses.

If the hardware is available, we will also run a physical grasp test that repeats the paper's Franka protocol. Its 87.50% success rate is the bar to beat.