Robots trained in simulation often fail on real hardware because their networks latch onto cues that only exist in the simulator, such as a rendering artifact, a fixed light direction, or a background that happens to correlate with success. A popular remedy is to make the robot's visual representation keep only what predicts future reward and discard everything else. This is called value-directed feature selection: two observations that are worth the same to the agent get the same internal code. The flaw is that "what predicts reward" is judged inside the simulator, and a simulator shortcut that predicts reward there will be kept as task-relevant. This note proposes to fix the selection criterion rather than the data. We keep a visual feature only if one shared value predictor built on it works equally well across many randomized versions of the simulator, varying rendering and physics together. The real world is then treated as one more version.
Concretely, a small trainable attention query sits on top of a frozen pretrained image encoder and picks which image patches the policy and value function see. It is trained by reinforcement learning (RL) with an added penalty. When the agent's prediction error on future reward is noticeably worse in some simulator variants than in others, the attended features are penalized. Unlike existing augmentation-consistency methods, this criterion does not demand that predicted values stay identical across variants. That demand is wrong once friction or mass changes, because the true value changes too. It only demands that the same mapping from features to value holds everywhere.
Core Claim
Value-directed abstraction is necessary for sim-to-real transfer but not sufficient. On its own it selects simulator shortcuts whenever they are reward-predictive in training. Replacing "predicts value" with "predicts value with a single predictor that is equally accurate across simulator variants" removes those shortcuts. It does so without the background-motion assumption used by MIIR, and without the fixed-dynamics assumption used by TRP. It should therefore improve transfer under joint visual and physical shift, and especially in the cases where a spurious cue is value-correlated within individual simulator instances.
Position Relative to Prior Work
Three papers motivate the proposal by showing what goes wrong. RL-ViGen benchmarks visual RL across five axes of shift and finds that augmentation-based methods collapse under changes in camera viewpoint and scene structure. REALM builds a real-to-sim-validated manipulation benchmark and finds that even large vision-language-action models break under spatial and behavioral perturbations. Both suggest that current representations match surface statistics instead of selecting what matters for the task.
Five lines of work supply the selection machinery and constrain the design.
Attentive Feature Aggregation (AFA) trains a cross-attention query over patches of a frozen pretrained encoder using imitation learning. It reports that this tripled out-of-distribution success for VC-1 on MetaWorld (about 15% to about 45%) and held 75% versus 17.5% on a real SO-101 arm with 20 distractors. We reuse its architecture but train the query with RL and an invariance penalty. AFA's queries learned from demonstrations have no notion of value and were never tested in closed-loop RL.
VCR and ROUSER show that value- and Q-predictive objectives beat reconstruction-based representations. VCR reports 39.7% versus 33.7% IQM against SPR on Atari 100K. Both measure value relevance in a single training distribution, which is exactly the criterion we argue is exploitable by shortcuts.
Denoised MDPs separates observation signals by controllability and reward relevance (628 versus Dreamer's 519 on RoboDesk). Its taxonomy does not have a category for signals that are reward-relevant only inside the simulator. That category is our target.
MIIR and TRP come closest. Both enforce that return-related predictions agree across perturbed views. MIIR perturbs low-variation pixels, and by its own account this heuristic fails on dynamic backgrounds. TRP requires identical truncated-return predictions across augmentations, which presumes that dynamics and rewards do not change. Our penalty differs in kind: it constrains the error of a shared predictor across environments, not the agreement of its outputs. That is what lets physics randomization enter the training set without contradicting the objective.
HIB infers privileged physics such as friction and terrain from proprioceptive history with an information bottleneck, and it climbs 0.6 m real stairs where baselines fail. We borrow its idea of a short proprioceptive-history encoder, so that physics-driven value differences can be explained by history instead of being forced onto visual shortcuts. HIB itself is limited to low-dimensional inputs.
IAAC shows that giving a critic the full privileged state hurts, because structured irrelevant signals act like a "noisy TV," and that a return-informative subset helps. This constrains our design: we do not feed the critic full simulator state.
Research Program
Each training episode is drawn from simulator variant $e$. A variant randomizes textures, lighting, camera pose within a range, distractor objects (including moving ones), friction, mass, and actuator latency. Within each variant the agent observes RGB frames and a short proprioceptive history.
The observation passes through four components. A frozen pretrained encoder (VC-1 or MAE, the "local-first" encoders where AFA worked) produces patch tokens. A learned query attends over those tokens. A small 1D-convolution encoder summarizes proprioception. The concatenated feature $\phi(o)$ feeds an actor and a critic $Q_\theta$ that are shared across all variants.
The critic and the query are trained on
$$\mathcal{L} = \frac{1}{|E|}\sum_{e\in E} \ell_e(\phi,\theta) + \lambda\,\mathrm{Var}_{e\in E}\big[\ell_e(\phi,\theta)\big],$$
where the symbols are as follows. $E$ is the set of variants in the current batch. $\ell_e$ is the mean squared temporal-difference error of $Q_\theta(\phi(o),a)$ on transitions from variant $e$, meaning the gap between predicted value and the one-step bootstrapped target. $\lambda$ sets the strength of the invariance pressure.
A spurious cue lowers $\ell_e$ only in the variants where it happens to correlate with reward, so it raises the variance and is pushed out of the attention. A genuine cue, such as the gripper–object relation, lowers the error everywhere. Physics changes legitimately shift the targets, but the proprioceptive branch lets the shared critic absorb those shifts, so they do not inflate the variance. The actor is trained by standard soft actor-critic on top of $\phi$.
The analysis component tracks how much attention falls on annotated patch categories (robot, target object, distractors, planted cue) over the course of training. Comparing that allocation with and without the penalty is the most direct test of the claim.
One known weakness is sparse reward. When most targets are zero, $\ell_e$ is small everywhere and the penalty carries no signal, which mirrors the bisimulation collapse described in Towards Robust Bisimulation Metric Learning. We will use dense simulator rewards and n-step targets, and we will report sparse-reward tasks separately rather than hide them.
Baselines
The naive reference point is DrQ-v2 trained with the same domain randomization over rendering and physics. RL-ViGen identifies this method as sample-efficient but brittle out of distribution, so it shows what randomization alone achieves.
The second baseline isolates the effect of pretrained features. It uses the same frozen VC-1 encoder with global average pooling and soft actor-critic, under the same randomization.
The key ablation is identical to the proposal with $\lambda = 0$: an AFA-style query trained purely by RL gradients, which is value-directed selection without invariance. Everything except the penalty is shared, including the encoder, proprioceptive branch, randomization set, number of environment steps, and a hyperparameter search of equal size (we will fix the number of trials per method in advance). If this ablation matches the full method, the core claim is false.
The strongest existing competitors are MIIR and TRP. Both target the same visual-generalization question with return-based consistency objectives, and both are run on DMControl-GB with their published settings plus our physics randomization. MIIR reports a 22% higher average return than SGQN and SVEA on video-hard. It also reports up to 3.9x better performance on PegBox. The file gives no absolute return for MIIR, so we will reproduce it under our compute budget and report both our reproduction and its relative claim.
On RL-ViGen we compare against PIE-G, which reports 89% Habitat success versus 42% for DrQ, and against SRM, which reports an episode return of 45 under CARLA weather shifts while other methods drop to near zero. For camera-view shifts the file has no reported numeric result for SGQN, only that it alone held performance on "hard" camera shifts. We will include SGQN and report what we measure.
Beating these baselines means two things. First, a statistically significant improvement (over 10 seeds) on held-out variants that combine visual and physical shift. Second, no loss on the training distribution.
For real-robot reference, AFA's 75% on SO-101 comes from imitation learning, so it is a different setting. We cite it as context, not as a direct competitor.
Why not just…?
Why not just randomize harder? Domain randomization is the default, and with enough variety it may remove most shortcuts. The evidence against relying on it alone comes from two sources. AFA's real-robot results show standard pretrained-encoder policies falling to 17.5% under real distractors. RL-ViGen shows augmentation failing on structural shifts. Still, randomization is what our method consumes as input, so the honest test is the DrQ-v2 baseline and the $\lambda = 0$ ablation, both trained on identical randomized variants. If either matches the full method, more randomization is the cheaper answer and we will say so.
Why not just train AFA's query with the RL loss? This is the most serious threat. RL gradients already push the query toward value-relevant patches, and that may be enough. Our argument is that value-relevance measured in training includes shortcuts. The planted-cue diagnostic below is designed to settle this directly: it creates a cue that is value-predictive within variants but not across them. If the $\lambda = 0$ query ignores the cue anyway, the penalty is unnecessary.
Why not just require consistent return predictions across augmentations, as TRP does? TRP's assumption that dynamics and rewards are identical across augmentations breaks as soon as physics is randomized. Forcing equal predictions then penalizes correct value differences. The physics-shift experiments compare TRP with and without physics randomization against our method. We expect TRP either to degrade when physics varies or to need physics held fixed, which would leave the dynamics gap open.
Why not just distill from a privileged-state teacher? Two-stage teacher-student distillation is standard in locomotion. However, HIB reports that it produces conservative policies, and IAAC shows that full-state critics suffer from noisy-TV effects. Neither addresses pixel-level shortcuts. We include an asymmetric critic with full privileged state as a secondary baseline on the manipulation tasks to check this.
Proposed Experiments
The decisive experiment is a planted-shortcut test on DMControl and MetaWorld. In each training variant we render a small colored patch whose hue correlates with the variant's friction coefficient, and therefore with achievable return. The correlation sign is flipped across a subset of variants, and the patch is fully decorrelated at test time. We expect value-directed selection without the penalty to put measurable attention mass on the patch and to lose return when it is decorrelated. We expect the full method to put near-zero mass on it. If both behave the same, the core claim fails.
The broad evaluation covers three settings. On DMControl-GB video-hard with held-out friction and mass, we report average return. On RL-ViGen we evaluate the Robosuite, Habitat, and CARLA domains along its camera-view and scene-structure axes, reporting success rate or return, to test whether invariance-driven selection moves the "structural wall." On RoboDesk we test whether the method handles dynamic distractors, the case where MIIR's frequency heuristic is known to fail; DMDP's 628 is the published reference.
For real transfer we take two routes. First, we evaluate policies in REALM, whose simulated task progression correlates with real Franka progression at $r = 0.921$, to estimate the real-world effect without a full hardware campaign. Second, if hardware is available, we deploy a pick-and-place task on an SO-101 arm with 20 distractors mirroring AFA's protocol, reporting success over at least 20 trials against the $\lambda = 0$ ablation.
Across all settings we sweep $\lambda$ and the number of variants per batch. We expect gains to grow with variant diversity and to vanish when variants share their spurious cues, a prediction that follows from the variance penalty being blind to cues constant across $E$. We also report wall-clock cost, because the penalty adds only per-variant loss bookkeeping, not extra networks.
References
- How to Learn Domain-Invariant Representations for Visual Reinforcement Learning: An Information-Theoretical Perspective
- What Effects the Generalization in Visual Reinforcement Learning: Policy Consistency with Truncated Return Prediction
- RL-ViGen: A Reinforcement Learning Benchmark for Visual Generalization
- REALM: A Real-to-Sim Validated Benchmark for Generalization in Robotic Manipulation
- Attentive Feature Aggregation or: How Policies Learn to Stop Worrying about Robustness and Attend to Task-Relevant Visual Cues
- Value-Consistent Representation Learning for Data-Efficient Reinforcement Learning
- Learning Robust Representations with Long-Term Information for Generalization in Visual Reinforcement Learning
- Denoised MDPs: Learning World Models Better Than the World Itself
- Bridging the Sim-to-Real Gap from the Information Bottleneck Perspective
- Informed Asymmetric Actor-Critic: Leveraging Privileged Signals Beyond Full-State Access
- Towards Robust Bisimulation Metric Learning