Robots trained in simulation fail in the real world for two reasons that are usually studied separately. The real world looks different: lighting, textures, camera angle and background all shift. It also moves differently: friction, mass, damping and motor lag shift. Methods that make a policy robust to bad physics, in the strongest form robust optimization against worst-case physics, almost all assume the policy sees clean state vectors such as joint angles and object positions. Methods that make a policy robust to appearance assume the physics stays fixed. A robot that sees pixels and meets unfamiliar physics gets neither guarantee.
This note proposes a way to train one pixel-based policy that is robust to both, at close to the cost of ordinary randomization. The policy itself only sees images, but during simulated training we also have a value estimator (the critic) that sees the true physical state. That critic can decide which of several possible next physical states is most damaging. We can therefore pick a worst-case next state using physics alone, render only that one state into an image, and randomize how it looks. The expensive part of worst-case training, querying many futures, happens in cheap physics space. The expensive part of visual training, rendering, happens once per step.
Core Claim
The closest method is EWoK (Bring Your Own (Non-Robust) Algorithm to Solve Robust MDPs by Estimating The Worst Kernel). At each step it simulates $N$ candidate next states, reweights them toward low value, and resamples one. This turns any ordinary RL algorithm into one that solves a robust Markov decision process, meaning an RL problem where an adversary may move the transition probabilities within a Kullback-Leibler (KL) ball around the simulator's. The reweighting has the closed form
$$ \tilde p(s' \mid s,a) \propto p_0(s' \mid s,a)\,\exp\!\big(-V(s')/\beta\big), $$
where $p_0$ is the nominal simulator kernel, $V$ is the current value estimate, and $\beta$ is a temperature set by the KL radius. EWoK was tested only on state-based Cartpole and DeepMind Control tasks. Its authors note that $N$ queries per step is prohibitive inside visual renderers, and they suggest learned world models as the remedy.
Our claim is that a world model is unnecessary in the common asymmetric training setup. If $V$ is a privileged critic over true simulator state, as in the asymmetric actor-critic used by World Model-based Perception for Visual Legged Locomotion, then the weights above need no images at all. Physics stepping in MuJoCo with saved and restored simulator states is cheap. Rendering is not. So the worst-case kernel costs $N$ physics steps plus one render, and it uses the true simulator, so it carries none of the compounding prediction error that afflicts visual world models (LaDi-WM: A Latent Diffusion-Based World Model for Predictive Manipulation).
On top of this physical adversary, the selected state is rendered under randomized appearance. The image encoder is regularized to map nominal and randomized renders of the same state to nearby features, following Robust Domain Randomization for Reinforcement Learning. The result is a single training loop in which the physical adversary sits in state space and appearance invariance sits in feature space.
The empirical question is whether this split works when both shifts happen at once. The Distracting Control Suite - A Challenging Benchmark for Reinforcement Learning from Pixels found that visual distractions compound non-linearly with each other. Nobody has measured whether visual and physical shifts compound with each other too, or whether a method built for one quietly breaks the other.
Research Program
The method has three parts. The first is a stochastic nominal kernel. DeepMind Control dynamics are deterministic, so each of the $N$ candidate next states is generated with a small fresh perturbation of mass, friction and damping plus action noise; this gives $p_0$ real spread for the reweighting to act on. The second is the resampling step above, scored by a critic that receives the true state. The third is a pixel actor trained with SAC on the rendered observation of the resampled state, plus the feature-distance penalty between a nominal render and a randomized render (backgrounds, colors, camera pose) of that same state.
The analysis has two parts. We measure wall-clock cost per environment step as $N$ grows, separating physics time from render time, to check the claim that robustness becomes nearly free. We also test the link to theory. Robustness and Regularization in Reinforcement Learning shows that robustness to transition shifts is equivalent to a penalty on the size of the value function. That paper and Slaoui et al. supply two different penalties, one on value and one on features, which the file notes have never been combined. We will report whether the two interfere during training, for example whether the feature penalty flattens the actor in a way that undoes worst-case-aware behavior. If they do interfere, that is a substantive finding about why the two literatures stayed apart.
A deliberate limitation is that the physical adversary protects the underlying state-transition process, not the observation process. Robustness to camera or sensor shift comes only from randomization and the feature penalty, with no worst-case guarantee. Extending the adversary to observations, in the spirit of Robust partially observable Markov decision process, is left out on purpose so that the core claim can be tested cleanly.
Baselines
The naive reference point is pixel SAC with data augmentation trained on the nominal simulator, with no physical or visual randomization. It shows how much each shift costs.
The strongest competitor is joint uniform randomization with a memory-based policy. It combines the recurrent dynamics randomization of Sim-to-Real Transfer of Robotic Control with Dynamics Randomization with visual randomization and the same feature penalty. It is the strongest because it already sees perturbed physics during training, whereas our method sees only nominal-plus-noise physics. On real Fetch puck-pushing, Peng et al. report a success rate of 0.89 ± 0.06 for this approach in its state-based form. For our pixel benchmark under combined physical and visual shifts, no reported result was found on Lacuna. It will get the same encoder, the same SAC backbone and the same number of environment steps. Its randomization ranges will be tuned with the same search budget as our KL radius, and those ranges must cover the test grid. Beating it means a higher worst-decile return over the joint test grid, with no more than a small loss in nominal return.
The second strong competitor is EWoK applied naively to pixels: the $N$ candidates are scored by a critic that sees only images, so every candidate must be rendered. This shows whether the privileged critic matters for quality as well as speed.
The third is a learned-world-model version of EWoK, in which an RSSM (a recurrent latent dynamics model, as in WMP) proposes the $N$ candidates. This is exactly what the EWoK authors suggested.
Two ablations isolate the ingredients: the physical adversary without the feature penalty, and the feature penalty without the physical adversary. Under matched step budgets, the full method should beat both ablations on the joint grid. If it beats them only on their own single-shift grids, the combination adds nothing.
Why not just…?
Why not just randomize both physics and appearance uniformly and use a recurrent policy? This is the real threat. It is cheap, proven on hardware, and trains on the shifted physics directly. The evidence against it is indirect: Gadot et al. report that EWoK often beat dynamics randomization even though randomization trained on the perturbed environments, and Adversarial Diffusion for Robust Reinforcement Learning reports that its adversarial diffusion method beat DR-PPO under large parameter shifts. Both results are state-based. Our worst-decile comparison on the joint grid settles the pixel case directly. If uniform randomization matches us there, we will report that the adversary is unnecessary once vision is in the loop.
Why not just put a view-synthesis adapter in front of a nominal policy? AnyCamVLA: Zero-Shot Camera Adaptation for Viewpoint Robust Vision-Language-Action Models raises LIBERO success under unseen viewpoints from 67.9% to 94.5% by warping test images back to the training camera. However, it needs known camera calibration, and it cannot correct friction or mass. Our physics-only test condition, which has no visual shift, shows this directly: any adapter leaves the nominal policy's physical brittleness untouched.
Why not just build the worst-case kernel inside a learned world model, as the EWoK authors propose? That route generalizes beyond simulators that can save and restore their state, which is a real advantage. The cost is that the adversary then exploits model errors rather than real physics. The world-model baseline measures exactly this, in both robust return and step cost. Our method is preferable only where simulator state cloning exists, which covers MuJoCo and most robotics simulators.
Proposed Experiments
The main testbed is the Distracting Control Suite on Cartpole, Walker-walk and Walker-run, where we add the physical perturbations EWoK used: mass, gravity, joint damping and torso length. Training uses nominal physics plus small noise and the suite's training distractions. Testing sweeps a full grid that crosses physical parameter shifts with distraction intensity, including held-out background videos and camera poses. We report three metrics: nominal return, mean return over the grid, and worst-decile return over the grid, which is a CVaR-style summary. CVaR, conditional value at risk, is the average over the worst outcomes.
The first decisive check is the interaction test. For each method we fit an additive model of return loss in physical shift and visual shift, then measure the leftover interaction term. This tells us whether coupled shifts compound and whether our method reduces the compounding. The second check is cost: steps per second and render calls per step against naive pixel EWoK as $N$ varies over 4, 8 and 16. We expect render calls to stay at two per step, one nominal and one randomized render for the feature penalty, regardless of $N$. The third check is nominal return on Walker-run, where EWoK showed a drop, to measure how much over-conservatism we pay.
As a stretch goal, we would run a real-robot pushing task in the style of Peng et al. using camera input. It would be compared only against joint uniform randomization, with the same number of trials and objects of altered contact, as in that paper's chip-bag condition.
References
- Bring Your Own (Non-Robust) Algorithm to Solve Robust MDPs by Estimating The Worst Kernel
- World Model-Based Perception for Visual Legged Locomotion
- LaDi-WM: A Latent Diffusion-Based World Model for Predictive Manipulation
- Robust Domain Randomization for Reinforcement Learning
- The Distracting Control Suite - A Challenging Benchmark for Reinforcement Learning from Pixels
- Robustness and Regularization in Reinforcement Learning
- Robust partially observable Markov decision process
- Sim-to-Real Transfer of Robotic Control with Dynamics Randomization
- Adversarial Diffusion for Robust Reinforcement Learning
- AnyCamVLA: Zero-Shot Camera Adaptation for Viewpoint Robust Vision-Language-Action Models