A Slot for the Unknown: Factored Latent Blocks for Synthetic-to-Real Pose Estimation

Models trained on rendered images often fail on real photographs. They pick up rendering quirks and never see real lighting, real sensor noise, or real surface wear. The usual fixes either repaint whole images to look like the other domain, or push all features toward a single domain-invariant vector. Neither says where the gap lives. Neither gives the real-world differences a designated place to go, so those differences leak into the quantities the model is supposed to predict.

We propose a different split. We train an encoder whose latent space is cut into separate multi-dimensional blocks: one block for object pose (a 3D rotation, which cannot be flattened into a single number without tearing), and one block each for identity, lighting, and material or texture. The key addition is a residual block that is only allowed to carry what the simulator never produced. A simulator provides the supervision that disentanglement methods normally lack, because we can render pairs of images that differ in exactly one known factor. Unlabeled real images are then reconstructed through the same blocks, with the pose block held to consistency rules. Real-only variation therefore has somewhere to go other than the pose block. The two-sentence version: simulator interventions supply free paired supervision for a block-structured latent space, and an extra real-only block soaks up the sim-to-real gap so it does not corrupt pose. We test whether this beats strong baselines that are equally invariant but unstructured, on the synthetic-to-real LineMod pose benchmark.

Core Claim

We claim that sim-to-real error in visual pose estimation is largely leakage: real-image variation outside the simulator's range gets encoded into the pose features. Leakage can be reduced by two things together. The first is a latent space factored into multi-dimensional blocks trained with simulator-intervention pairs. The second is a residual block that absorbs real-only variation during reconstruction of unlabeled real images. The claim is falsifiable in two ways. If an unstructured encoder trained on the same intervention pairs with a contrastive invariance loss matches our pose error, the structure adds nothing. If removing the residual block leaves the error unchanged, the "slot for the unknown" idea is wrong.

Where This Sits

The disentanglement literature supplies the method, and it also names the obstacle. Towards a Definition of Disentangled Representations defines disentanglement as a split of the latent space into subspaces, each acted on by one subgroup of world transformations. We adopt that definition directly, with the rotation group acting on the pose block. Learning disentangled representations via product manifold projection builds this with an encoder, non-linear projectors onto disjoint submanifolds, and weak supervision from pairs sharing a factor. It reports a 99.2% BetaVAE score on SmallNORB versus 87.9% for Ada-GVAE, and shows that multi-dimensional blocks beat scalar ones. Dual Swap Disentangling swaps latent blocks between paired samples to enforce modularity. CITRIS proves that multi-dimensional causal factors are identifiable when intervention targets are known, but tests only on synthetic renders.

All four papers say the same thing about the obstacle: disentanglement needs pairs or known interventions, and these are hard to get. A simulator removes that obstacle, because every render comes with its intervention labels. Independent Subspace Analysis for Unsupervised Learning of Disentangled Representations constrains our design. It shows that rotation-invariant Gaussian priors make the axes within the latent space unidentifiable, and that raising $\beta$ over-prunes dimensions. We therefore use its non-Gaussian $L^p$-nested priors between blocks instead of a plain $\beta$-VAE. Disentangling by Subspace Diffusion motivates the multi-dimensional pose block. It reports that scalar VAEs fail on 3D rotation factors, where its method reaches about 0.02 radians of subspace angle error. None of these papers addresses a domain gap.

The sim-to-real literature motivates the problem, and its methods become our baselines. Unsupervised Pixel-Level Domain Adaptation with Generative Adversarial Networks (PixelDA) repaints synthetic images to look real with a GAN. Sim-To-Real via Sim-To-Sim (RCAN) maps randomized and real images to a clean canonical render. VR Goggles for Robots translates real images to simulated ones with CycleGAN plus a temporal-stability loss. All three operate on whole images and leave the latent factors unstructured. Their own reports note hallucinated artifacts and limits under structural shift.

Affine Transport for Sim-to-Real Domain Adaptation aligns sim and real distributions with one closed-form Gaussian map. Generalizable Domain Adaptation for Sim-and-Real Policy Co-Training aligns a single shared latent space with unbalanced optimal transport. Both align the representation as one piece. Neither partitions it or reserves capacity for variation the simulator lacks. Understanding Domain Randomization for Sim-to-real Transfer sets a limit we take seriously: its guarantees assume the real world lies inside the simulator's randomization range. The residual block is designed for the case where that assumption fails.

Research Program

The encoder maps an image to an intermediate feature. Separate projection heads then map that feature onto blocks $z = (z_{pose}, z_{id}, z_{light}, z_{mat}, z_{res})$. The pose block is a 6-dimensional continuous rotation representation that decodes to $SO(3)$, the group of 3D rotations. A decoder reconstructs the image from the concatenated blocks, and a pose head reads only $z_{pose}$.

Using the LineMod CAD models, we render intervention pairs where exactly one factor changes: pose, object, light direction and color, material and texture, or camera noise. Training on these pairs uses three signals. The first is a swap loss, in the style of Dual Swap and Fumero et al.: exchanging every block except the intervened one should reconstruct the partner image. The second is supervised pose regression on renders. The third is the $L^p$-nested prior across blocks. On renders, $z_{res}$ is forced toward a fixed value, so the simulator never teaches the model to use it.

Unlabeled real images from the target split enter only through reconstruction and two self-checks on the pose block. The first check is equivariance: rotating a real image in the image plane by angle $\theta$ must change $z_{pose}$ by the corresponding rotation about the camera axis. This is a genuine group action on a known subgroup, and it is available on real images without labels. The second check is invariance: photometric edits must leave $z_{pose}$ unchanged. Because $z_{pose}$ is pinned by these checks and $z_{light}$, $z_{mat}$ are pinned by the simulator's ranges, the cheapest way to reconstruct real-only appearance is through $z_{res}$. That routing is the mechanism under test.

We also measure the gap block by block. For each block we compute the distance between sim and real encodings (a kernel two-sample statistic). We also measure a leakage score: how much $z_{pose}$ moves when only $z_{res}$ is varied through decoding and re-encoding. We then ask whether these block-wise scores predict per-object pose error on real images.

Baselines

The naive reference is a pose regressor with the same backbone, trained on uniformly randomized renders with no adaptation. It shows how large the raw gap is.

The strongest published competitor on this benchmark is PixelDA (Unsupervised Pixel-Level Domain Adaptation with Generative Adversarial Networks). It reports a mean angle error of 23.5° on synthetic-to-real LineMod, against 56.58° for DANN and 53.27° for DSN. It is the right competitor because it uses the same unlabeled real images and the same labeled synthetic data, and it closes the appearance gap by pixel translation rather than latent structure. We will quote the published figure. We will also rerun PixelDA with our backbone and render set, under a matched GPU-hour and hyperparameter-search budget, so that any gain cannot come from better renders. Beating it means a lower mean angle error than both our rerun and 23.5°, over three seeds.

RCAN (Sim-To-Real via Sim-To-Sim) is the canonical-image competitor. We will train it on our randomized renders and apply it to real test images before the naive regressor. No LineMod result for RCAN is reported on Lacuna; its reported numbers are grasp success rates.

The most important baseline is an unstructured invariance encoder. It is trained on exactly the same intervention pairs, but with a contrastive loss that pulls together pairs sharing pose and pushes apart pairs differing in pose. It uses the same real-image equivariance and invariance checks, and has no blocks, no reconstruction, and no residual block. No reported result was found on Lacuna for this baseline. It isolates whether the block structure matters at all.

Affine Transport (Affine Transport for Sim-to-Real Domain Adaptation) will be applied to the naive regressor's penultimate features, fitting the Gaussian map on unlabeled real features. No LineMod number for it was found on Lacuna.

Three ablations isolate our ingredients: removing $z_{res}$, replacing each multi-dimensional block with scalar coordinates, and dropping real-image reconstruction.

Why not just…?

Why not just use domain randomization? Randomization is already inside our render pipeline, so the question is whether it is enough alone. Sim-To-Real via Sim-To-Sim reports that standard randomization reached only 33–37% zero-shot grasp success, against 70% for its canonical mapping. Understanding Domain Randomization for Sim-to-real Transfer ties randomization's guarantees to the real world lying inside the randomized range. The naive baseline measures this directly on LineMod.

Why not just learn invariance contrastively on the same simulator pairs? This is the real threat. The simulator pairs, not the blocks, might carry all the benefit. Our argument is that a contrastive encoder has no designated place for real-only variation, so that variation leaks into pose. The unstructured invariance baseline, run with identical data and real-image checks, settles this. If it matches us, we will conclude that the block structure is unnecessary for accuracy and that its value is limited to diagnosis.

Why not just translate real images to simulation, or simulation to real? PixelDA, RCAN, and VR Goggles do exactly this and are rerun as baselines. Their own papers report hallucinated artifacts, flicker requiring extra losses, and failure under structural shift. Pixel translation also gives no per-factor diagnosis, which is the other half of this proposal.

Why not just use a scalar disentangled VAE? Disentangling by Subspace Diffusion and Learning disentangled representations via product manifold projection show scalar latents break on rotations. The scalar-block ablation tests whether that matters for transfer as well as for disentanglement scores.

Proposed Experiments

The first experiment is a controlled sim-to-sim study, where leakage can be measured exactly. We render LineMod objects with a second renderer that has a lighting and reflectance model absent from training, and treat those renders as "real." Because ground-truth factors are known here, we can report how much of the held-out nuisance variation is predictable from $z_{pose}$ versus $z_{res}$, measured by $R^2$ as in CITRIS. The decisive check: with the residual block, held-out nuisance should be predictable mainly from $z_{res}$. Without it, that nuisance should become predictable from $z_{pose}$, and pose error should rise with it.

The main experiment is synthetic-to-real LineMod under the PixelDA protocol. We report mean angle error and classification accuracy over three seeds against all baselines and ablations. As a pre-registered prediction, the block-wise gap and leakage scores should rank objects by real-image pose error, measured by Spearman correlation across the 11 objects. A method whose diagnostics do not track its failures would undercut the interpretability claim even if accuracy improves.

As a secondary transfer check, we apply the same encoder recipe to DREAM-real camera-to-robot pose estimation. There, Markerless Camera-to-Robot Pose Estimation via Self-Supervised Sim-to-Real Transfer reports 85.9% AUC using known kinematics and CAD-based rendering. We do not expect to beat it without kinematics. The question is how much of that gap a kinematics-free factored encoder closes relative to DREAM's roughly 68%.