Robots often need to know an object's 6-DoF pose, meaning its position in space (translation) and its orientation (rotation), and they increasingly learn this from simulated images alone. The standard way to make a network trained on synthetic images work on real camera images is domain randomization: the simulator renders each scene with random textures, lights, backgrounds and distractor objects, so the network learns to ignore those nuisances. This note starts from one empirical hint. The visual randomizations that help a network localize an object appear to differ from the ones that help it orient the object. In the one controlled study that measured both, realistic directional lighting and shadows mattered most for position, while textures and geometric distractors mattered most for orientation (Benchmarking Domain Randomisation for Visual Sim-to-Real Transfer).
That study used two completely separate networks, so it could not ask the question that matters in practice. Suppose one shared image encoder feeds a rotation head and a translation head. Does randomization that strips away a cue one head needs hurt the other head through the shared features? And can we avoid this by choosing which randomized rendering each head's loss is computed on? We propose to render every training scene twice with the same object poses: once "physically faithful," with realistic lighting and shadows, and once "aggressively randomized," with random textures and distractors. The translation loss is computed on the first rendering and the rotation loss on the second. A consistency term ties the two views' shared features together. The claim is that this per-head routing gives better zero-shot real-world accuracy than feeding both heads the same randomized mixture, at the cost of a single encoder.
Core Claim
In a shared-backbone pose network trained only on synthetic data, the rotation and translation heads compete for different visual cues. Uniform randomization forces a compromise that neither head would choose. Assigning each head its own randomization stream, with both streams rendered from identical scene geometry, should reduce real-world rotation error and translation error together on cluttered benchmarks. It should do so without the memory and latency cost of two separate encoders. The new element is not decoupling itself, and not randomization itself. It is making the randomization distribution a per-output design choice, and testing whether that choice interacts with a shared representation.
Grounding in Prior Work
Three decoupled architectures supply the design space we build on. PoseCNN splits pose into segmentation, translation by pixel-wise voting for the object center plus depth, and quaternion rotation from cropped features. It was trained with real images and render-and-paste compositing, not zero-shot randomization, so it tells us the architecture works but not how it behaves under domain shift. FINet keeps separate rotation and translation branches but lets them exchange features. This motivates our shared encoder over fully disconnected networks, although FINet works on point clouds and never faces a visual reality gap. Li et al. map translation to 2D pixel offsets plus scale change and regress rotation relative to the object center. We adopt this parameterization so that the translation head's depth estimate depends on apparent size and shading cues, which the faithful stream preserves.
On the randomization side, Alghonaim & Johns motivate the proposal. They found that moving from flat-shaded to high-fidelity rendering cut $xy$-position error from 4.37 cm to 0.53 cm and orientation error from 13.0° to 3.08°, and that directional lights reduced position error by about 68.4%. They also found that a dataset with only 25% high-quality images performed nearly as well as an all-high-quality one. That "25% rule" is also the strongest threat to our idea, because simple mixing may already capture most of the benefit. Their evaluation used four isolated primitive shapes, which is why we move to cluttered benchmarks.
BOP Challenge 2020 and CosyPose constrain the proposal. They showed that physically based renderings plus heavy color, texture and lighting augmentation reach 69.8% Average Recall, which beats render-and-paste. So the faithful stream must be physically based rendering, not a toy renderer. Structured Domain Randomization (52.2 vs 24.0 AP on KITTI Hard) suggests that context-consistent placement matters, so both streams share physically settled object piles rather than random floating objects.
Finally, DeepCLR argues for the opposite design: a single dual-quaternion output that removes the need to balance two losses. We treat it as a competing hypothesis to be tested, not dismissed.
Research Program
The work proceeds in three stages.
Stage one: measure the sensitivity asymmetry. The asymmetry must hold beyond primitives before any method is built on it. We take the BlenderProc4BOP setup and render YCB-Video, T-LESS and Occluded-LINEMOD training scenes. We vary one randomization factor at a time: texture randomization, distractor density, light direction and shadow presence, background images, and camera noise. For each setting we train a shared-encoder network with decoupled heads and report real-test rotation and translation errors separately. If lighting and texture factors do not affect the two errors differently on these objects, the routing idea has no basis. In that case we stop and publish the negative measurement.
Stage two: build the routed model. Each training scene is rendered as a pair $(x^f, x^r)$ with identical poses, where $x^f$ is the faithful rendering and $x^r$ the aggressively randomized one. A shared encoder $E$ feeds a translation head $h_t$ and a rotation head $h_R$. The training objective is
$$ \mathcal{L} = \ell_t\big(h_t(E(x^f)), t\big) + \ell_R\big(h_R(E(x^r)), R\big) + \lambda \,\big\| E(x^f) - E(x^r) \big\|^2 . $$
Here $t$ and $R$ are the ground-truth translation and rotation. $\ell_t$ is an L1 loss on image-center offset and log-depth, and $\ell_R$ is a symmetry-aware point-matching loss in the style of PoseCNN's ShapeMatch loss. The last term asks the encoder to produce the same features for both renderings of a scene, which keeps the representation shared, and $\lambda$ sets its weight. At test time only one real image is used and both heads run on it. The consistency term is what lets a real image, which is neither stream, map to features both heads can use. We therefore also run a variant with $\lambda = 0$.
Stage three: diagnose the mechanism. We measure the gradient cosine between the two heads' losses on the shared encoder under uniform and routed randomization, to check whether routing reduces conflict. We also probe which image regions each head relies on, for example shadows versus silhouettes, using simple occlusion tests on real images.
Baselines
The naive reference point is a single coupled head that regresses translation and quaternion from one shared feature vector. It is trained on the union of both renderings with the same randomization for all outputs, at matched image count, epochs and tuning budget. Every other method must beat it. Otherwise decoupling adds nothing under randomization on these benchmarks.
The strongest existing competitor is CosyPose's single-view pipeline trained on BOP physically based renderings with its heavy augmentation. BOP Challenge 2020 reports 69.8% Average Recall for CosyPose over its datasets. CosyPose reports YCB-Video AUC of ADD-S of 89.8 with one view. The file does not state whether that YCB-Video figure used real training images. We therefore rerun CosyPose's refiner on exactly our synthetic images with no real training data, using the same detector crops and a comparable GPU-hour budget, and treat the published numbers as an upper reference. Beating it means higher BOP Average Recall on at least two of YCB-V, T-LESS and LM-O under this strictly synthetic protocol. A smaller but still meaningful result would be matching it while being substantially faster, since our model is a single-pass network with no iterative refinement.
The mixture baseline isolates the key ingredient. It uses the identical shared-encoder, two-head network and the identical paired renderings, but both heads see both streams: each loss is computed on $x^f$ and $x^r$ alike, including the 25%-faithful mix that Alghonaim & Johns found near-optimal. The "two independent networks" baseline trains separate encoders, each on its preferred stream, which is Alghonaim & Johns' design applied to cluttered objects. We add a dual-quaternion single-head variant following DeepCLR, and a reversed-routing control in which rotation gets the faithful stream and translation the randomized one. If routing works for the reason we claim, reversed routing should hurt. If it merely adds data diversity, reversal should not matter. The proposal is supported only if routed training beats the mixture baseline by more than seed variance (five seeds) on both error components, and reversed routing does worse than both.
Why not just…?
Why not just train two independent networks, as Alghonaim & Johns did? This is a genuine competitor on accuracy, because two encoders cannot interfere with each other at all. Its costs are doubled memory and inference time, and the loss of shared geometric features. FINet reports that letting branches exchange information helped registration, reaching 0.383° rotation error on ModelNet40, but that evidence comes from point clouds. We run this baseline directly. If it matches routed training on accuracy, the contribution shrinks to efficiency, and we will say so.
Why not just mix faithful and randomized renderings for both heads? This is the most serious threat. The "25% rule" shows that mixing already recovers most of the benefit of high fidelity for isolated primitives. Routing is only worth its complexity if the heads really pull the shared encoder in different directions. The mixture baseline and the gradient-conflict measurement settle this. Equal accuracy together with low measured conflict would falsify our premise.
Why not just use photorealistic rendering without randomization? BOP Challenge 2020 indicates that its winning physically based pipeline still relied on heavy augmentation. We include a faithful-only model anyway, because on texture-less T-LESS parts, texture randomization may matter less than on textured YCB objects.
Why not just unify the outputs with a dual quaternion? DeepCLR's motivation, avoiding loss balancing, is legitimate. It was, however, shown on point-cloud fine registration within a limited search radius, not on single RGB images under domain shift. The dual-quaternion variant is run with the same data and budget. If it matches decoupled heads, the case for decoupling under randomization weakens.
Proposed Experiments
All models train on the same count of BlenderProc renderings per dataset, with each pair counting as two images for the baselines. They are evaluated with no real training images on the BOP test splits of YCB-Video, T-LESS and Occluded-LINEMOD. We report BOP Average Recall and also separate translation error in centimeters and rotation error in degrees, because the hypothesis is about the two components.
As a physical check, we deploy the best model on a robot arm with an eye-in-hand camera to estimate poses of YCB objects. We compare against the roughly 1.5 cm single-view ceiling reported by Domain Randomization for Active Pose Estimation. That paper's active multi-view averaging is complementary, so we also report routed-model results with its four-view aggregation.
The decisive checks come in three steps. Stage one must show the asymmetry on real cluttered data. Routed training must beat the mixture baseline on both error components. Reversed routing must underperform. If any of the three fails, the claim that randomization should be assigned per output is wrong, and the stage-one measurements are the result worth reporting.
References
- Benchmarking Domain Randomisation for Visual Sim-to-Real Transfer
- PoseCNN: A Convolutional Neural Network for 6D Object Pose Estimation in Cluttered Scenes
- FINet: Dual Branches Feature Interaction for Partial-to-Partial Point Cloud Registration
- DeepIM: Deep Iterative Matching for 6D Pose Estimation
- BOP Challenge 2020 on 6D Object Localization
- CosyPose: Consistent multi-view multi-object 6D pose estimation
- Structured Domain Randomization: Bridging the Reality Gap by Context-Aware Synthetic Data
- DeepCLR: Correspondence-Less Architecture for Deep End-to-End Point Cloud Registration
- Domain Randomization for Active Pose Estimation.