Locating the Sim-to-Real Gap in Surface Physics: Pose Estimation Through an Intervenable Intrinsic-Map Bottleneck

A model trained on simulated images usually gets worse when deployed on real photographs. Today we can measure how much worse, but rarely why. We propose to make the reason measurable. The model is a 6D object pose estimator, meaning a model that recovers an object's 3D position and orientation from an image. Instead of mapping pixels straight to a pose, it is forced to first predict a small set of physically defined per-pixel maps: surface normals (which way each point of the surface faces), depth, object-surface coordinates, albedo (the surface's intrinsic color with lighting removed), and a specularity/roughness map (how shiny the surface is). The pose is computed only from these maps plus a deliberately narrow leftover channel. The simulator supplies exact labels for every map at no cost. A detail of existing pose benchmarks then makes the idea work: real test photos come with ground-truth poses and CAD models, so we can render the true geometric maps for real images too.

That combination enables a direct experiment. On real images, we replace one predicted map with its rendered ground truth and measure how much pose accuracy recovers. The recovery tells us how much of the sim-to-real gap that physical factor is responsible for. If these attributions are faithful, they should also tell us where to spend effort. That means re-rendering training data with more variation only in the failing factor, or adapting only that channel at test time, and this targeted spending should beat uniform randomization at equal cost. The proposal treats interpretability as a measurement tool for sim-to-real transfer rather than as an end in itself. Surface appearance modeling supplies the vocabulary of the bottleneck.

Core Claim

The claim is that a pose estimator constrained to reason through simulator-supervised surface maps gives up little accuracy relative to an unconstrained network. In return, it gives a causal, per-factor decomposition of its own sim-to-real gap, and that decomposition predicts which targeted fix works. The accuracy cost is to be measured, not assumed. The decomposition is checked against what actually happens when we act on it. The claim is wrong if the attributions do not predict the outcome of targeted interventions. It is also wrong if a black box with post-hoc probes yields equally faithful attributions.

Grounding in Prior Work

Three lines of work motivate the proposal.

BOP Challenge 2020 on 6D Object Localization showed that physically based rendering, meaning ray-traced images that model light scattering and inter-reflection, is what finally let deep networks trained on synthetic data beat classical geometric matching. The winning CosyPose reached 69.8% Average Recall. The resulting estimators are opaque regressors, however, and give no account of which surface property causes residual failures.

Robust Visual Sim-to-Real Transfer for Robotic Manipulation decomposed domain randomization, the practice of randomly varying textures, lighting and color during simulated training. It found that realistic textures beat procedural noise, and it tuned randomization on a cheap proxy task. Its ResNet-18 policy is still a black box, and its randomization budget is chosen by search rather than by diagnosing what breaks.

Manipulation as in Simulation showed that real depth sensors fail on shiny, dark and transparent surfaces. It repaired them with an RGB-conditioned network that "cleans reality to match simulation," lifting a kitchen task from 0/30 to 26/30. That network is itself an uninspectable filter.

Taken together, these papers show that surface appearance matters a great deal for transfer, and that the field has no way to say which aspect of it matters for a given failure. They also disagree on whether appearance should be modeled faithfully (BOP, AmbientCG textures) or filtered out (clean depth). Our attribution experiment speaks directly to that disagreement, factor by factor.

Two lines of work supply the method.

Concept Bottleneck Models route predictions through human-named concepts so that a person can overwrite a wrong concept at test time. Our "concepts" are dense physical maps rather than semantic labels. Their test-time overwrite becomes our oracle intervention.

CONDA adapts concept bottlenecks to distribution shift at test time using unlabeled target data. It aligns concept-score statistics, re-weights the final predictor, and adds residual concepts, recovering 28% accuracy on Waterbirds. CONDA's concepts are text-derived and come from CLIP pseudo-labels, and it never touches simulation. We reuse its score-alignment idea, applied per physical channel and gated by our attribution.

Two papers constrain the design.

The first is Beyond Concept Bottleneck Models. It shows that black-box latents can be probed post hoc and edited for intervention using as little as 0.5% of the training data. This is the strongest argument that no architectural bottleneck is needed, so we treat it as a baseline rather than a footnote. The same line of work, together with CONDA, warns that strict bottlenecks lose accuracy unless a residual pathway exists. Residual pathways in turn leak information that bypasses the concepts, so residual width becomes a controlled variable here.

The second is This Looks Like That... Does it?. It shows that interpretable intermediate units can latch onto confounds. Our bottleneck channels are therefore validated against rendered ground truth rather than trusted by name.

On the appearance side, SOPHY disentangles shape, color and physical material into separate latent codes for generation. It is the closest precedent for factorizing surface properties, but it leaves the factors as continuous latents and never studies downstream perception. OrienText and the older finding of Oxholm et al. that normal fields are more stable than raw appearance motivate making normals a first-class channel.

Research Program

The architecture is a standard image encoder that predicts, per pixel inside each detected object crop, five map groups. The geometric group is normals, depth, and 3D object-surface coordinates. The appearance group is albedo and specularity/roughness. The model also predicts a residual feature map whose channel count $r$ we sweep from zero upward. A pose head sees only these maps. It is a small network followed by a Perspective-n-Point solver, the classical algorithm that recovers pose from 2D-3D point correspondences. We are explicit that the pose head is not linear. The interpretability we claim is mediation through physically defined, independently verifiable quantities, not a transparent readout.

Training uses the BlenderProc PBR renders distributed with BOP, re-rendered with the extra passes the renderer can export for free: normals, depth, coordinates, albedo, and roughness.

The central quantity is the gap closed by intervening on channel $k$. Let $c_k$ be the predicted map, $c_k^\ast$ its rendered ground truth on the real image, and $\mathrm{AR}$ the benchmark's Average Recall. The per-channel recovery is

$$\Delta_k = \mathrm{AR}\big(\text{pose head with } c_k \leftarrow c_k^\ast\big) - \mathrm{AR}\big(\text{pose head with predicted maps}\big),$$

reported as a fraction of $\Delta_{\text{all}}$, the recovery when every available channel is replaced.

Geometric channels have real ground truth on all BOP datasets. Albedo has real ground truth only where textured models exist (LM-O, YCB-V), and can be approximated there by rendering the model texture. Specularity has no real ground truth, so for that channel we can only intervene indirectly, by substituting the simulator's value for the matched material. We report this as a weaker estimate. We also report pairwise interventions to check whether the $\Delta_k$ add up or interact.

The repair stage turns attributions into actions in two ways. The first is targeted re-rendering. A fixed budget of additional PBR images varies only the factor with the largest $\Delta_k$ on a held-out real validation split, for example lighting and specular parameters if the normal and albedo channels are the ones failing. The second is channel-gated test-time adaptation. CONDA-style statistic alignment is applied on unlabeled real images to the attributed channel alone, rather than to the whole representation.

Baselines

The naive reference is the same encoder and pose head trained on the same PBR data with no bottleneck: the maps are replaced by an unconstrained feature map of equal size, and training uses the same steps and augmentation. This is the "do nothing" point that defines the accuracy cost of the bottleneck. The bottleneck is acceptable if it stays within a stated margin of this model's real-image AR. We pre-register a two-point AR loss as the tolerance.

The strongest existing method is CosyPose as reported in BOP Challenge 2020, with 69.8% Average Recall across the challenge datasets. The file does not specify which training data produced that figure, so we will rerun the public CosyPose pipeline on PBR-only training for our dataset subset to get a matched number. We do not expect to beat it in raw AR with a lighter encoder. The relevant comparison is that CosyPose cannot produce $\Delta_k$ at all. We therefore also apply our repair actions to it, using uniform randomization at the same added budget, to see whether targeted repair on our model closes more of its own gap than uniform repair closes for CosyPose.

The ablation that isolates the key ingredient is a multitask model. It predicts the same five maps as auxiliary outputs from a shared trunk, but the pose head reads the trunk rather than the maps. It uses the same supervision, data and tuning budget. If this model matches our accuracy, as we expect, the comparison then rests on whether its maps can serve as faithful intervention points.

The final baseline is the post-hoc probe approach of Beyond Concept Bottleneck Models. Linear or small convolutional probes are fit from the black box's features to each map, and interventions are made by editing features toward the ground-truth map. Probes are trained on the same simulated labels, and the tuning budget is matched. Beating these baselines means two things. Our $\Delta_k$ must predict real repair outcomes with higher rank correlation than probe-derived attributions. And targeted repair guided by our attributions must outperform uniform randomization by more than seed variance at equal rendering cost.

Why not just…?

Why not just use heavy domain randomization on a black box? It works well: 93% real success in Robust Visual Sim-to-Real Transfer. It may also match our raw accuracy. But the same paper found that some randomizations, such as camera and color extremes, hurt, and had to search for good settings with a proxy task. Our repair experiment is a direct test of whether diagnosis beats search. Uniform and proxy-tuned randomization at the same added image budget are the comparison arms. If uniform randomization matches targeted repair, the practical case for the bottleneck reduces to inspection alone, and we will say so.

Why not just add the intrinsic maps as auxiliary losses without forcing the pose through them? This is the most serious threat to any accuracy claim, and we concede that point in advance. The multitask ablation will likely be at least as accurate. What it cannot guarantee is that its predicted maps are what the pose depends on. Overwriting an auxiliary output changes nothing downstream, so its $\Delta_k$ is zero by construction. Its maps can only report channel error, not channel responsibility. The experiment that settles this asks whether per-channel error alone, which multitask gives, predicts repair outcomes as well as per-channel recovery, which only the bottleneck gives. If it does, the bottleneck is unnecessary.

Why not just probe a black box post hoc and edit its features? Beyond Concept Bottleneck Models makes this cheap. Its weakness here is faithfulness: editing a feature toward "correct normals" need not remove the information the network actually uses, and leakage through other directions is the documented failure of residual pathways. We test this directly by comparing probe-edit recovery against true recovery from our bottleneck on the same images, and by checking which attribution better predicts repair. If probe attributions are equally predictive, we will recommend probes.

Why not just discard appearance and use cleaned depth? Manipulation as in Simulation shows that this needs an RGB-conditioned cleaning network, without which success is 0/30. The original BOP paper, BOP: Benchmark for 6D Object Pose Estimation, reports roughly 30% drops for methods under occlusion (LM versus LM-O). Our attribution answers this question empirically rather than by argument. If the geometric channels account for nearly all of $\Delta_{\text{all}}$ and the appearance channels for almost none, the "filter appearance out" position is vindicated for pose estimation. That is a useful outcome either way.

Proposed Experiments

We will use the BOP datasets LM-O, T-LESS, YCB-V and TUD-L. These span textureless industrial parts, textured household objects and heavy occlusion. Training is on BOP PBR renders, and evaluation is on real test images. Metrics are the BOP Average Recall and its components: Visible Surface Discrepancy, which scores pose error only on visible surface depth, and the symmetry-aware surface and projection distances.

The first experiment draws the accuracy-interpretability curve. We plot real-image AR against residual width $r$ for the bottleneck model. Alongside it we measure leakage, defined as AR when all maps are replaced by ground truth but the residual is shuffled across images. This shows where the model stops depending on its physical channels.

The second experiment produces the gap decomposition. We report $\Delta_k$ for each channel and each dataset, along with pairwise interaction terms. We expect normals and coordinates to dominate on textureless T-LESS, and albedo and specularity to matter more on YCB-V. We commit to reporting whichever pattern appears.

The third experiment is the decisive check. Attributions are computed on a held-out real split. The extra rendering budget is then allocated to the top-ranked factor, the second-ranked factor, a random factor, and uniform randomization. We test whether real-image AR improvements on the test split follow the attribution ranking. Channel-gated test-time alignment is run under the same ranking protocol.

We would consider the proposal successful under three conditions. Repair aimed at the top-attributed factor must beat uniform randomization and repair aimed at the lowest-attributed factor across seeds. The bottleneck's attributions must predict repair outcomes better than multitask channel errors and probe-edit attributions do. And the accuracy cost must stay within the pre-registered margin.

A secondary transfer check uses the Hammer dataset from Manipulation as in Simulation. It contains specular and transparent objects where depth sensors fail. There we test whether the specularity channel's attribution identifies those objects as the dominant failure source. No reported pose result on Hammer was found in the file, so this check has no external reference number.