Lighting-Consistency Abstention for Pose Estimators Trained on Synthetic Data

Robots that pick up machined or plastic parts often rely on a pose estimator, a network that predicts an object's 3D rotation and position from a camera image. These networks are usually trained only on rendered images and then run on real ones. When the real scene contains a highlight the renderer never produced, the network can return a confidently wrong pose and gives no sign that it has failed. We propose a training-free check that tells the system when to abstain, meaning decline to act and instead re-image, ask a human, or fall back to a safer routine. The check asks one physical question: if the predicted pose were correct, could the observed shading be explained by smooth, distant lighting falling on the object's known surface normals?

Smooth lighting is well described by a few low-order spherical harmonics, a Fourier-like basis on the sphere of directions. Fitting such a model takes a nine-coefficient least-squares solve per hypothesis. The usual way to verify a pose is to render the object and compare it with the image, and specular glints corrupt that comparison. Our check splits the comparison by angular frequency. The low band carries the pose evidence, and glints mostly fall into the high band. We then add a second test: a correct pose should send those glints to a coherent set of reflection directions. The result is a selection function for continuous 6D pose that keeps working where synthetic training is weakest, on shiny surfaces.

Core Claim

The spherical-harmonic low-pass bottleneck is usually treated as a defect. Spec-Gaussian shows that low-degree harmonics in 3D Gaussian Splatting cannot represent specular lobes, so optimization compensates with spurious "floater" primitives. Neural-PIL makes the same point for spherical Gaussians, which blur windows and light panels. We use the bottleneck on purpose, as a filter that separates two kinds of disagreement between a rendered pose hypothesis and a real image:

  • Disagreement explainable by smooth lighting on the hypothesized normals. This kind is benign.
  • Disagreement that is not. This kind indicates a wrong pose, unless the leftover residual organizes coherently in reflection-direction space, which is what a real glint on a correctly posed object does.

Our claim is that a selection function built from this split gives a better risk-coverage trade-off than (a) plain render-and-compare residuals and (b) learned or sampling-based confidences. Risk-coverage means how often the system is wrong on the cases it accepts, as a function of how many cases it accepts. We expect the advantage to grow with the fraction of the object covered by highlights.

Research Program

Inputs. For a pose hypothesis $T$ from any estimator, we render the CAD model to obtain, for each visible pixel $p$:

  • the surface normal $n_p$;
  • the mirror-reflection direction $r_p$;
  • a visibility mask, intersected with the detector's instance mask to drop occluders.

The observed crop is roughly linearized with an assumed inverse gamma. The constant and scale terms of the lighting fit absorb unknown exposure and white balance, which CHROMA identifies as a source of violated photometric assumptions.

Score 1: low-band misfit. Texture-less objects such as those in T-LESS have nearly constant albedo, so under a correct pose the diffuse shading is close to a function of the normal alone. We define

$$s_{\text{low}}(x,T)=\min_{c\in\mathbb{R}^{9}}\frac{\sum_p \rho\big(I_p-c^\top Y_{\le 2}(n_p)\big)}{\sum_p \rho\big(I_p-\bar I\big)}$$

where:

  • $I_p$ is the linearized intensity at pixel $p$;
  • $Y_{\le 2}$ stacks the nine spherical harmonics of degree at most two;
  • $\bar I$ is the mean visible intensity;
  • $\rho$ is a Huber loss that down-weights a minority of outlier pixels.

A wrong pose assigns wrong normals to pixels. The resulting shading is then not a smooth function on the sphere, and $s_{\text{low}}$ rises. For textured objects (YCB-Video), we divide by the albedo from the CAD texture before fitting.

Score 2: reflection-space incoherence. Huber down-weighting alone would let a wrong pose hide its misfit by calling it "glints." So we test the outlier pixels in a second step. We bin the residuals $I_p-c^\top Y_{\le 2}(n_p)$ by reflection direction $r_p$ on a coarse spherical grid, and measure the within-bin variance relative to total residual variance, giving $s_{\text{spec}}(x,T)$.

  • Under a correct pose, a glint comes from one bright source, so pixels sharing a reflection direction share a residual and the score is low.
  • Under a wrong pose, outliers scatter across bins.

Decision rule and calibration. The selection function is $g(x)=1$ iff $s_{\text{low}}\le\tau_1$ and $s_{\text{spec}}\le\tau_2$. We pick the thresholds on a small labeled real calibration split using a risk-control procedure in the spirit of Act or Defer. That work shows average accuracy can hide errors concentrated in the accepted set, and it argues for explicit error budgets. We will report both the guaranteed bound and the realized risk on held-out scenes.

Symmetry. The check respects object symmetry automatically. Symmetric poses of a symmetric CAD model produce identical normal maps, so it does not penalize the per-image ambiguity that BOP-D documents. Unlike BOP-D's depth-based pruning, it needs no depth, which matters because active depth sensors fail on specular parts, as Pose Estimation of Specular and Symmetrical Objects reports.

Cost. Estimators such as CosyPose already render for refinement, so the added cost is one normal buffer and a nine-dimensional solve.

Known failure modes. Cast shadows from clutter and inter-reflections are high-frequency in normal space and will inflate $s_{\text{low}}$ on correct poses. Neural-PIL avoids this issue only by assuming a single bounce. We will measure this effect rather than assume it away.

Baselines

Naive reference: no abstention. The estimator answers every detected instance at 100% coverage. This is the setting the BOP protocol scores; in BOP Challenge 2020 the winning CosyPose reached AR = 69.8%. Every selective method is drawn as a risk-coverage curve that starts at this point.

Strongest competitor: plain render-and-compare residual. We render the textured CAD model at the predicted pose under the training renderer's default lighting and score the pose by a robust (Huber) photometric distance to the crop. We also run a version that adds a silhouette term. This is the obvious verification any practitioner would write and directly exploits CosyPose's renderer, which is why it is the strongest competitor. It receives the same Huber parameter search and the same real calibration split as our method.

Other existing signals. We run three further comparisons, all wrapped around the same two base estimators: CosyPose, and the augmented-autoencoder pipeline of Multi-Path Learning.

  • Codebook similarity. The cosine similarity from the augmented-autoencoder pipeline, which that work computes but never calibrates as an abstention score.
  • Learned confidence head. A small head regressing pose error, trained on the same BlenderProc physically-based synthetic data as the estimator.
  • Sampling-based disagreement. The spread of refined poses from ten perturbed initializations, with Monte Carlo dropout as a variant, at a matched wall-clock budget.

Ablations. Four ablations isolate the key ingredient:

  • a full-band fit, using high-degree harmonics so glints are "explained" too;
  • low-band misfit without the reflection-space test;
  • the reflection-space test alone;
  • an edge-and-silhouette consistency score in the style of Hu et al.

What counts as winning. Our method wins if it has lower area under the risk-coverage curve than the render-and-compare residual on real test scenes. The margin must exceed the spread across five calibration/test scene splits, and it must widen in the top tercile of highlight fraction. No reported result found on Lacuna for selective risk-coverage of any 6D pose estimator, so all selective numbers will be produced in this study.

Why not just…?

Why not just use the render-and-compare residual the refiner already computes? This is the real threat, and the proposal stands or falls on it. Our hypothesis is that a glint on a correctly posed shiny part produces a large residual against a synthetic render, so the plain score rejects correct poses. Meanwhile a wrong pose on a matte region can render plausibly, so the plain score accepts wrong ones. The highlight-stratified comparison tests this directly. If a well-tuned robust residual matches our score even in the high-highlight tercile, the lighting decomposition adds nothing, and we would report that as the finding.

Why not just train a confidence head on synthetic data? The head learns what errors look like in renders. The sim-to-real literature argues that networks latch onto synthetic shortcuts (Freeman et al.), and that selection functions calibrated in-distribution become overconfident under shift (Selective Classification for Deep Neural Networks). A physics-based test has no synthetic-trained parameters to drift. We will measure each head's synthetic-to-real calibration drop alongside ours.

Why not just use sampling disagreement or Monte Carlo dropout? Disagreement detects instability, not wrongness. An estimator that is consistently fooled by a highlight that mimics an edge will agree with itself. Hu et al. describe exactly this mechanism, and they report only 2.2% ADD on small nuts. Sampling also multiplies inference cost, against fast BOP methods that run at 0.63 s per image. The matched-budget comparison will show whether disagreement catches the highlight-driven errors.

Why not just check silhouettes and edges, as Hu et al. do? Silhouettes are ambiguous for symmetric and texture-less parts, and highlights create false edges. Edge consistency is complementary, not competing, so we also report the combination with our score.

Proposed Experiments

Mechanism test in simulation. We use BlenderProc4BOP physically-based renders of T-LESS and YCB-Video objects. Material roughness is swept from matte to near-mirror, and poses are perturbed at controlled angular errors. This shows how each score separates correct from wrong poses as glossiness increases. The prediction is that render-and-compare separation collapses at low roughness while ours degrades slowly.

Main real evaluation. We use the real T-LESS and YCB-Video test sets with the estimators trained only on synthetic data. The error label is a pose whose symmetry-aware surface distance (MSSD) exceeds the BOP threshold. We report:

  • area under the risk-coverage curve;
  • risk at 80% coverage;
  • coverage at a 5% risk budget.

All results are split by scene and stratified by per-instance highlight fraction, estimated as the fraction of pixels whose intensity exceeds the low-band fit by a fixed margin. Two published reference points anchor the base estimator's unconditional accuracy: CosyPose's YCB-Video AUC of ADD-S, 89.8 with one view in the CosyPose paper, and its BOP 2020 AR of 69.8%.

Failure-mode measurement. We measure false rejections attributable to cast shadows and inter-reflection, using heavily cluttered T-LESS scenes. If such scenes are numerous and the release terms permit, we will add the industrial metallic parts of Hu et al. Otherwise we will capture a small set of real shiny parts with fiducial ground truth.

Decisive check. On the high-highlight tercile of real instances, our score must beat the tuned render-and-compare residual in area under the risk-coverage curve across all five splits. Its realized risk must also stay at or under the calibrated budget on held-out scenes. If either condition fails, the central claim is wrong.