Robots trained in simulation often fail on real hardware because the real world's friction, mass, and delays differ from the simulator's. The standard fix, domain randomization, trains the policy across many randomly perturbed simulators so that the real robot looks like one more variation. That only works if the policy can tell, from its recent history, which variation it is in. Contrastive predictive coding is a natural way to teach a network that skill. It is a self-supervised method that learns features by asking a model to pick the true future from a set of decoys. Used naively, though, it learns the wrong thing. The easiest way to pick the true future is to look at the current position of objects or at the background texture. The model then ignores the physics, which is the only part that actually needs identifying.
This note proposes a simple, simulator-specific fix. Draw the decoys from counterfactual rollouts: reset the simulator to the exact current state, replay the exact same actions, and change only the physics parameters, re-rendering each rollout with fresh random textures. The true future and its decoys then share the same starting state and the same commands. The only reliable way to tell them apart is to have inferred, from the past, how this particular world responds to pushes. Separately randomizing textures within the true pair removes appearance as a cue. The resulting representation should carry the dynamics information a zero-shot policy needs and little else. We will test this on visual pushing with a Franka Panda arm, both in simulation and on the real robot.
Core Claim
Let $c_t$ be the output of a recurrent history encoder over past images and actions, $s_t$ the simulator state at time $t$, $a_{t:t+k}$ the next $k$ actions, and $z_{t+k}$ the encoding of the image observed $k$ steps later. The positive $z^{+}_{t+k}$ comes from the episode's true physics parameters $\theta$. The $N-1$ negatives $z^{j}_{t+k}$ come from resetting to $s_t$, executing $a_{t:t+k}$ under fresh parameters $\theta_j$ drawn from the randomization prior, and rendering with independent random textures. We minimize
$$ \mathcal{L} = -\mathbb{E}\left[\log \frac{\exp f(z^{+}_{t+k}, c_t, a_{t:t+k})}{\exp f(z^{+}_{t+k}, c_t, a_{t:t+k}) + \sum_{j=1}^{N-1} \exp f(z^{j}_{t+k}, c_t, a_{t:t+k})}\right], $$
where $f$ is a learned score in the style of an action-conditioned bilinear critic. Standard InfoNCE draws its negatives from the marginal distribution of futures. Here the negatives come from the conditional distribution $p(z_{t+k}\mid s_t, a_{t:t+k})$, so the same argument as in Representation Learning with Contrastive Predictive Coding turns the loss into a lower bound on a conditional quantity:
$$ I\big(z_{t+k};\, c_t \,\big|\, s_t, a_{t:t+k}\big) \;\ge\; \log N - \mathcal{L}. $$
This is the information the history carries about the future beyond what the current state and actions already reveal. Under randomized physics, that is exactly the information that identifies the dynamics. The quantity the shortcut exploits, the instantaneous state, contributes zero to it by construction.
Two consequences follow. First, the objective removes the shortcut that ContraBAR: Contrastive Bayes-Adaptive Deep RL reports, where action-conditioned contrastive prediction encoded only current-state features. Second, it reconciles the two camps the research file sets against each other. Unsupervised Feature Learning for Manipulation with Contrastive Domain Randomization argues for invariance to randomized factors. Understanding Domain Randomization for Sim-to-real Transfer and Sim-to-Real Transfer of Robotic Control with Dynamics Randomization argue for history-based identification of them. The objective gets both: textures vary inside the positive pair, which produces invariance, and physics varies across the negatives, which forces identification.
Relation to Prior Work
Three papers supply the method.
- Representation Learning with Contrastive Predictive Coding introduced the encoder, the autoregressive context, and the InfoNCE loss. We change only where the negatives come from.
- ContraBAR added action conditioning and proved that contrastive future prediction yields sufficient statistics for Bayes-adaptive control. It drew negatives from other tasks, worked across episodes in meta-reinforcement learning, and stayed in simulation. We keep its action-conditioned predictor but use same-state counterfactual negatives, in a strict zero-shot setting with hardware deployment.
- Contrastive Domain Randomization (CDR) (Unsupervised Feature Learning for Manipulation with Contrastive Domain Randomization) is the closest work. It also intervenes in the simulator, rendering one physical state under two textures. It contrasts only the same timestep and is explicitly invariance-only. The authors list "extending this invariance to physical parameters like friction and mass" as future work. We do the complementary thing for physics: we demand sensitivity to physics, not invariance, and we predict across time.
Two papers motivate the proposal by showing the gap from the privileged-information side.
- CRA-Loco (Contrastive Representation Learning for Robust Sim-to-Real Transfer of Adaptive Humanoid Locomotion) distills privileged terrain features into a proprioceptive actor with InfoNCE, aligning current features.
- The Historical Information Bottleneck (HIB) (Bridging the Sim-to-Real Gap from the Information Bottleneck Perspective) aligns a history summary to privileged parameters with positive-only cosine similarity.
Both target the parameter vector itself rather than its consequences for future observations. Neither uses negatives that hold state and action fixed.
Two papers constrain the design.
- The Latent Markov decision process analysis of Understanding Domain Randomization for Sim-to-real Transfer shows that memory is required. It also shows that guarantees fail when reality lies outside the randomization support. We therefore test held-out parameter ranges explicitly.
- Contrastive Difference Predictive Coding reports that its temporal-difference variant is hyperparameter-heavy and struggles on visually complex control. We therefore use the simpler Monte Carlo form of the loss.
Research Program
Simulator setup. We build a counterfactual sampling layer on a MuJoCo manipulation environment from robosuite using the Panda model, and on GPU-parallel ManiSkill3 where batched state resets are cheap.
Collecting counterfactual negatives. For an anchor timestep, the layer snapshots the full simulator state and draws $N-1$ physics parameter sets. Each set covers object mass, surface friction, joint damping, actuation latency, and observation noise, matching the parameters Sim-to-Real Transfer of Robotic Control with Dynamics Randomization found necessary. The layer then replays the logged actions for horizons $k \in \{2, 4, 8\}$ and renders each branch with Describable Textures Dataset textures and random lighting.
Agent. The representation is trained as an auxiliary loss alongside an asymmetric actor-critic in the style of Asymmetric Actor Critic for Image-Based Robot Learning. The critic sees simulator state. The actor sees only images and proprioception, passed through the recurrent context encoder. The same context also feeds a model-predictive-control planner, so we can compare against CDR on its own evaluation protocol.
Main risk: wasted negatives. Many counterfactual branches will be indistinguishable. A gripper not yet touching the puck reveals nothing about the puck's friction. In such branches the loss gives no signal rather than wrong signal, so wasted compute is the main risk, not a corrupted representation. We will measure how often branches are indistinguishable and test a contact-gated sampler that spends counterfactual negatives only where branches diverge. This connects to the "communicating" condition in the Latent MDP analysis: identification is possible only when the history visits informative states.
Reading the conditional bound. The conditional bound gives a diagnostic for free. Comparing the loss against a version whose context sees only $s_t$ estimates how much dynamics information the history has extracted.
Baselines
Naive reference: recurrent policy with randomization. This is the recurrent policy trained with dynamics and visual randomization and no auxiliary loss, following Sim-to-Real Transfer of Robotic Control with Dynamics Randomization. It is the do-nothing reference because that paper shows the recurrent policy alone performs implicit identification. It reported a real-world puck-pushing success rate of 0.89 ± 0.06 on a Fetch arm, against 0.67 for a feedforward policy with randomization. We cannot reuse that number directly because our robot and observations differ. We rerun the method with the same simulator, randomization ranges, network size, environment steps, and hyperparameter search budget as our method.
Strongest existing visual-invariance method: CDR. We use same-timestep texture contrast with standard negatives, following Unsupervised Feature Learning for Manipulation with Contrastive Domain Randomization. On its real Panda cube-pushing task under model-predictive control, it reported an average planning error of 7.59 cm, against 15.26 cm for its baseline. We reimplement CDR inside our pipeline with matched encoder and data. On the real planning task, beating it means lower error than our rerun of CDR when object mass and surface friction are changed. That is the regime CDR does not target, so it is where a gain must appear.
Strongest privileged-identification competitor: supervised parameter regression. The history encoder regresses the true parameters $\theta$, or aligns to a privileged embedding of them in the style of HIB (Bridging the Sim-to-Real Gap from the Information Bottleneck Perspective). It gets the same data and tuning budget.
Key-ingredient ablation: ordinary negatives. This is the identical action-conditioned predictive loss with negatives drawn from other episodes, as in ContraBAR. It isolates the counterfactual negatives as the single change. We will also report counterfactual negatives without texture re-rendering, to separate the invariance half from the identification half.
Success criterion. The method counts as better only if it beats the ablation and the regression baseline on held-out-range transfer, with confidence intervals over seeds and real trials that do not overlap.
Why not just…?
Why not just train a recurrent policy with domain randomization and let it identify the dynamics itself? This is the default, and it worked on a Fetch arm from state inputs. The evidence that it falls short with images is indirect. ContraBAR found that pure recurrent meta-learning performed poorly on image-based Panda reaching. Asymmetric Actor Critic for Image-Based Robot Learning reports that symmetric pixel-only training struggled to reach 40% in simulated Fetch picking. Our naive baseline will show whether the reward signal alone suffices in our task. If it matches our method at equal compute, the auxiliary loss is unnecessary and we will report that.
Why not just regress the simulator parameters from history, since the simulator knows them? This is the real threat, and we take it seriously. Our argument against it is specific. Regression must spend capacity on parameters that barely affect behavior. It is ill-posed when parameters trade off against each other, for example when heavier mass looks like higher friction. And it gives no signal about effects outside the parameter vector. The counterfactual loss targets only differences that change the future, and it treats physically equivalent parameter sets as equivalent. We test this directly in two ways. First, with deliberately redundant parameter sets, such as coupled mass and friction plus irrelevant distractor parameters. Second, under unmodeled real effects, such as an attachment that alters contact, like the chips packet in the Peng et al. experiment. If regression ties us in both settings, we will reframe the contribution as showing that counterfactual contrast is an annotation-free equivalent rather than an improvement.
Why not just use the ordinary predictive-coding negatives from ContraBAR? The authors themselves report that this objective can be minimized by encoding current-state features. Our conditional bound explains why: marginal negatives differ in object position, so position wins. The ablation above settles the question. As a mechanistic check, we will probe whether current object position is linearly decodable from the context and whether physics is. We expect position to dominate under ordinary negatives and physics under ours.
Why not just reconstruct future frames with a generative world model? ContraBAR reports that VariBAD's pixel decoder was computationally infeasible on its image-based Panda tasks. Reconstruction also spends capacity on randomized textures, which is exactly what we want discarded. We do not rerun a pixel model. We include a state-space forward-prediction baseline in the style of Not Only Domain Randomization: Universal Policy with Embedding System Identification as the generative counterpart where it is tractable.
Proposed Experiments and Decisive Checks
Simulation sweep. Pushing and lifting on the robosuite Panda run under three conditions: in-distribution randomization, a held-out range where mass and friction lie beyond the training range on both sides, and redundant-parameter randomization. We report success rate, identification speed, and the conditional-information gap.
- Identification speed is the number of history steps needed before a linear probe recovers the dynamics-relevant quantities. These are the effective friction-to-mass ratio and latency.
- The conditional-information gap is the loss of the full context minus the loss of a state-only context.
Real-robot test. A zero-shot Franka Panda pushing task with an Intel RealSense camera uses objects whose mass and friction are changed with added weights and surface sleeves. Two metrics are reported: CDR's Euclidean planning error under model-predictive control, and policy success rate over at least 20 trials per condition. A modest number of trials per method is honest about hardware limits, so we report binomial intervals.
Decisive check. The headline comparison is against the ordinary-negatives ablation on held-out physics ranges. If counterfactual negatives do not improve held-out transfer there, the core claim is false, regardless of in-distribution gains.
Secondary check. Compare against parameter regression under redundant and unmodeled dynamics. Here, a win confirms the "predict consequences, not parameters" argument, and a tie narrows it.
Cost accounting. We report the cost of counterfactual branching as wall-clock time and simulator steps, and compare all methods at equal simulator-step budgets so the extra rollouts are not a hidden advantage.
References
- Representation Learning with Contrastive Predictive Coding
- ContraBAR: Contrastive Bayes-Adaptive Deep RL
- Unsupervised Feature Learning for Manipulation with Contrastive Domain Randomization
- Understanding Domain Randomization for Sim-to-real Transfer
- Sim-to-Real Transfer of Robotic Control with Dynamics Randomization
- Contrastive Representation Learning for Robust Sim-to-Real Transfer of Adaptive Humanoid Locomotion
- Bridging the Sim-to-Real Gap from the Information Bottleneck Perspective
- Contrastive Difference Predictive Coding
- robosuite: A Modular Simulation Framework and Benchmark for Robot Learning
- ManiSkill3: GPU Parallelized Robotics Simulation and Rendering for Generalizable Embodied AI
- Asymmetric Actor Critic for Image-Based Robot Learning
- Not Only Domain Randomization: Universal Policy with Embedding System Identification