Free Ensembles from the Refinement Loop: Stochastic Iterate Averaging for Sim-to-Real Pose Refinement

Many 6D pose estimators work by "render and compare": they guess where an object is, draw a synthetic picture of its 3D model at that guess, compare the drawing to the real camera image, and correct the guess. The step repeats several times. These refiners are trained almost entirely on synthetic images, so the comparison is always between a clean rendering and a messy real photo. When the photo contains reflections, missing texture, or sensor noise the network never saw, each correction is pushed off by a fixed error. A deterministic loop repeats the same mistake on every pass. Separately, there is a known way to cancel errors of this kind: inject randomness into a network at test time and average several noisy predictions. That works, but it multiplies inference cost by the number of samples.

This note proposes using the refinement loop itself as the ensemble. Every iteration of a stochastic refiner sees the image under a fresh random perturbation, and the final pose is an average over the late iterates rather than the last one. The loop already pays for $K$ forward passes. A deterministic loop spends them repeating one systematic error, whereas a randomized loop turns them into $K$ partly independent samples at no added latency. The claim is testable: if the part of the sim-to-real error that changes with the random perturbation is large, iterate averaging should close a measurable share of the synthetic-to-real gap on hard industrial benchmarks at the same compute. If that part is small, the idea fails cleanly, and a direct measurement will show which case holds.

Core Claim

Iterative pose refiners trained on synthetic data carry error that is partly fixed (the same at every iteration) and partly perturbation-dependent (it changes when the input's low-level appearance is randomized). Randomizing appearance at every refinement step and averaging the trailing iterates on the rotation and translation manifold removes the perturbation-dependent part with no additional forward passes. We predict the gain is concentrated on textureless and reflective objects, where the synthetic-to-real gap is largest.

Concretely, let $T_k$ be the pose estimate after iteration $k$, $I$ the real image, $R(T_k)$ the rendering at that pose, and $\xi_k$ a fresh random draw of a perturbation layer. The refiner applies

$$T_{k+1} = \Delta_\theta\big(\xi_k(I),\, \xi_k(R(T_k))\big) \circ T_k, \qquad \bar T_K = \mathrm{mean}_{SE(3)}\{T_k : K_0 < k \le K\},$$

where $\Delta_\theta$ is the learned update network, $\circ$ composes poses, and the mean is translation averaging plus chordal quaternion averaging over the last $K-K_0$ iterates. Suppose the perturbation-dependent error at each iterate has variance $\sigma^2$ and correlation $\rho$ between iterates. Averaging $n=K-K_0$ iterates then shrinks that component to about $\sigma^2(1+(n-1)\rho)/n$, which is the standard variance of a mean of correlated samples. Measuring $\rho$ is therefore part of the work.

One design choice is specific to render-and-compare: the same random draw $\xi_k$ is applied to both the real image and the rendering. The refiner's job is to compare the two. A shared random restyling removes appearance as a usable cue for matching while leaving geometry intact, so the network must align shapes rather than textures.

Relation to Prior Work

The motivating evidence comes from the pose-refinement line. DeepIM introduced iterative render-and-compare with a flow-based matcher over 2 to 4 deterministic loops, raising LINEMOD 5cm 5° accuracy from 19.4% to 85.2% at about 12 fps. CosyPose carried this to multi-view, multi-object scenes. MegaPose made the refiner object-agnostic for novel objects, at about 50 ms per iteration. Coupled Iterative Refinement coupled recurrent flow with differentiable PnP, a solver that recovers pose from point matches, and reached AR 0.776 on T-LESS. FoundationPose scaled diffusion-textured synthetic data and added a learned hypothesis ranker. All five refine deterministically. They close the domain gap through better synthetic data or more iterations, never by making the iterations disagree on purpose. We keep their architectures and change only what each iteration sees and how the output is read off.

The method comes from the stochastic-averaging line. Network Randomization prepends a randomly re-initialized convolution layer, trains with a feature-matching loss between clean and randomized views, and averages actions over $M$ random draws at deployment. It reports, for example, a jump from 59.2 to 356.8 on Surreal Robotics with unseen textures. Random Self-Ensemble shows such averaging gains saturate near 10 passes, which happens to match typical refinement depths. Both papers pay $M$ extra passes for a single feedforward decision. We remove that cost by spreading the samples across iterations that already happen.

Two papers constrain the design. Iterative Partial Refinement injects noise inside an iterative diffusion loop and gains on constraint tasks, but at roughly 13× compute, and it degrades when re-noising is too aggressive ($\alpha=0.50$). We take this as a warning to keep perturbations at the appearance level and never re-randomize the pose state itself. Network Randomization also notes that randomization can destroy tasks that depend on color cues, so we expect and measure possible harm on strongly textured YCB-V objects.

Meta-Reflection and Discrete Diffusion VLA attack refinement latency in other ways, by distilling the loop into one pass or by using fewer parallel decoding steps. Ours differs in keeping the loop's cost fixed and changing what it produces.

Research Program

We build on two open refiners with published training pipelines: the CosyPose refiner, for seen objects on the BOP core datasets, and the MegaPose refiner, for novel objects. Each is retrained from its own synthetic PBR data with two additions. The first is a random-convolution layer in the Network Randomization style, applied with a shared draw to the rendered and observed crops. The second is a feature-matching loss that ties randomized and clean intermediate features together.

At test time, each refinement iteration draws a new layer, and the reported pose is the manifold mean of iterates after a short burn-in $K_0$. We then fix the number of network evaluations, for example 4, 8, or 12, and compare reporting the last iterate with reporting the trailing average, for both deterministic and stochastic refiners.

The analysis has three parts. First, an error decomposition: on frozen real images with ground truth, sample many draws at a fixed pose, and split the update error into its mean (the fixed bias) and its spread (the perturbation-dependent part). Do the same on held-out synthetic images, to test whether the extra real-image error sits mostly in the spread. Second, estimate the correlation $\rho$ across iterates, which predicts how much averaging can deliver. Third, run a latency audit in milliseconds per object on one GPU, including a variant that runs a small batch of draws in parallel inside each iteration. This tests whether width or depth is the better place to spend stochasticity.

Baselines

The naive reference point is the unmodified deterministic refiner, retrained on identical synthetic data with identical steps, reporting the final iterate. It shows what the loop does when it simply repeats itself.

The strongest published refiner on the hardest seen-object sets is Coupled Iterative Refinement, which reports BOP Average Recall of 0.776 on T-LESS and 0.893 on YCB-V, against 0.861 for CosyPose on YCB-V. Its cost is over 6 seconds per image according to BOP 2022. That report also gives the speed-accuracy reference: GDRNPP at 80.5 AR_C in 0.23 s per image, against a 2022 top of 83.7 AR_C and an ITODD score of 67.9. We will not retrain CIR. We cite its numbers and judge our CosyPose-based variant on the AR-versus-milliseconds curve. Beating it means reaching CIR-level T-LESS recall at a small fraction of its time, or improving recall on T-LESS and ITODD at equal time.

For novel objects, the competitor is the MegaPose refiner run deterministically with the same number of iterations and the same coarse initializations. FoundationPose is reported as first on the unseen-object BOP leaderboard with about 1.3 s initialization, but the file contains no per-dataset BOP AR for it. We therefore treat it as a published upper reference, not a re-run.

Two ablations isolate the key ingredient. The first averages the trailing iterates of the deterministic refiner. This tests whether averaging alone, without randomness, explains any gain, for instance by damping oscillation. The second applies a stochastic perturbation at each iteration but reports the last iterate. This tests whether training with randomization alone explains the gain. A third ablation uses independent rather than shared draws for the rendering and the real image.

All variants use the same synthetic data, detector, initial poses, total network evaluations, and hyperparameter search budget. Success means the full method beats both single-ingredient ablations on T-LESS and ITODD, with confidence intervals from three training seeds.

Why not just…?

Why not just run more deterministic iterations? If the excess real-image error were basin-of-attraction failure, extra steps would fix it. A fixed appearance-induced bias, by contrast, defines a wrong fixed point that more steps only converge to more tightly. DeepIM uses 2 to 4 loops, and its gains are known to plateau. Our AR-versus-iteration curves for the deterministic baseline, extended to 12 steps, settle this directly: if they keep climbing to match the stochastic variant, the proposal is unnecessary.

Why not just make the synthetic data better? This is the real threat. BOP 2022 reports PBR training within about 2.5 points of real-data training on average, and FoundationPose's diffusion texturing pushes further. However, that average hides the textureless and reflective sets (T-LESS, ITODD), which the file flags as where PBR struggles most. Our method also composes with better data rather than replacing it. We will report the per-dataset gap between PBR-only and PBR-plus-real training for both deterministic and stochastic refiners. If the gap left after good data is already within seed noise on T-LESS and ITODD, we will report that the idea has no room to help.

Why not just do test-time augmentation with M parallel passes? That is the Network Randomization recipe, and on a GPU a small batch may cost little extra wall-clock time. We include it as the width variant at matched evaluations and matched milliseconds. If it matches iterate averaging, the honest conclusion is that where the randomness is spent does not matter. Even then, the error decomposition and the shared-draw finding would stand as the contribution. Iterate averaging should win wherever the batch cannot be absorbed: on many objects per image, on embedded hardware, or when draws within one iteration are more strongly correlated than draws across iterations.

Proposed Experiments

The main experiment trains seen-object refiners on BOP PBR data and evaluates on the BOP core sets T-LESS, ITODD, LM-O, and YCB-V with the standard VSD, MSSD, and MSPD Average Recall. Novel-object refiners are trained on MegaPose's synthetic corpus and evaluated on YCB-V and the same BOP sets. Every result is plotted against both network evaluations and measured milliseconds per object.

The first decisive check is the error decomposition. If the perturbation-dependent part of the update error is a small share of the extra error on real images compared with synthetic images, the mechanism is falsified regardless of headline numbers. The second is the predicted pattern across datasets. Gains should be largest on T-LESS and ITODD, and negligible or slightly negative on texture-rich YCB-V objects, where randomization may erase useful color cues. A uniform gain would instead suggest a generic regularization effect. The third is the per-dataset synthetic-to-real gap with and without the method. Shrinking that gap on T-LESS and ITODD at unchanged latency is the outcome the proposal is built to deliver.

A secondary transfer test applies stochastic unmasking order with averaging across refinement steps to the 12-step decoder of Discrete Diffusion VLA. It targets that paper's reported 20.4% degradation under visual scene perturbation on LIBERO while holding its 14.53 Hz control rate fixed. This tests whether the principle carries beyond pose refinement to a different refinement loop, without changing the main claim.