Robots learn to open drawers, cabinets and trash cans from recorded experience. Most of that experience comes from a handful of objects that artists modeled by hand, with physics that someone tuned. Such an agent overfits to those few objects. Generative simulation can produce thousands of new objects cheaply, using language models to propose tasks and retrieve meshes, but their physics (hinge stiffness, damping, latches) is guessed and never checked. This note asks a concrete question. When an offline reinforcement learning agent (one trained only on a fixed log of past interactions, with no trial and error of its own) must work on objects it has never seen, is it better served by many cheaply generated assets with unverified physics or by a few carefully curated ones? And can the generator's own uncertainty about physics be turned into a training signal, so that the rough assets become safe to learn from?
The proposed mechanism is simple. For every logged trajectory on a generated asset, we replay the same recorded actions in the simulator under several plausible re-samplings of that asset's joint physics. How far the outcomes spread tells us how much a transition depends on physics the generator never verified. That spread enters the value update as a penalty, the same role an uncertainty penalty plays in pessimistic offline RL. The result is a controlled measurement of how fidelity trades against diversity for offline learning, together with a method that makes low-fidelity diversity usable.
Core Claim
The Generalization Gap in Offline Reinforcement Learning showed on Procgen that offline agents generalize to unseen levels far better when the same one million transitions are spread over 100,000 levels instead of 200. It also found that plain behavioral cloning often beats offline RL there. That study covered only discrete-action games and named continuous control as open. In parallel, ArtVIP hand-built 992 articulated assets with motion-capture-verified, position-dependent joint dynamics and warned that this curation does not scale. RoboGen scales, but it hallucinates geometry and physics.
We claim three things for articulated manipulation. First, at a fixed transition budget, offline generalization to unseen objects is governed mainly by the number of distinct training assets, as in Procgen. Second, the benefit of generated assets saturates, or reverses, as their unverified physics drives the learned value function toward dynamics that do not exist. Third, penalizing transitions by their sensitivity to resampled physics moves that saturation point outward. The result we care about is an exchange rate: how many generated assets are worth one curated asset, with and without the penalty.
Research Program
We first build a generated-asset corpus with a RoboGen-style pipeline. PartNet-Mobility and Objaverse objects are retrieved per category. A language model proposes each asset's joint parameters: stiffness, damping, friction, and whether a latch or detent exists. Instead of a single guess, the language model outputs a range for every parameter. That range defines the per-asset physics distribution we will resample. Textures are varied with CasTex so that visual diversity does not go to zero. All assets are exported to the same Isaac Sim / USD setting that ArtVIP uses, so the curated and generated assets share a simulator.
Next we collect offline datasets. Behavior is deliberately of mixed quality: motion-planner solutions, the same solutions with injected action noise, and failed or partial attempts. This matters because behavioral cloning is known to shine only on expert data, and the research file raises exactly this caveat about the BC result in The Generalization Gap in Offline Reinforcement Learning. Following that paper's protocol, we fix the total number of transitions (for example one million). We then sweep the number of distinct training assets on a log scale, from tens up to the size of the generated pool.
The method, physics-resampled replay, works as follows. For a logged transition $(s, a, r, s')$ on asset $j$, we draw $K$ physics samples $\xi_1, \dots, \xi_K$ from that asset's language-model-proposed range. For each sample we reset the simulator to $s$, apply the logged action $a$, and record the outcomes $(r_k, s'_k)$. No new policy is ever run. The action sequence stays exactly what the log contains, so the method remains offline with respect to behavior and uses the simulator only as a counterfactual physics engine. This is in the spirit of the counterfactual simulation in FQI-AIR, except that the uncertain part here is the physics of a generated asset rather than an exogenous variable.
The value target then becomes
$$y(s,a) = \frac{1}{K}\sum_{k=1}^{K}\big[r_k + \gamma V(s'_k)\big] - \beta\,\Gamma(s,a), \qquad \Gamma(s,a) = \mathrm{std}_k\big[r_k + \gamma V(s'_k)\big].$$
Here $V$ is the learned state value, $\gamma$ the discount, and $\beta$ a penalty weight. $\Gamma$ is the spread of one-step returns across physics samples. The target plugs into IQL or CQL unchanged.
The design follows Is Pessimism Provably Efficient for Offline RL?, whose penalty $\Gamma_h$ measures how poorly the data covers a region. Ours measures instead how much a transition depends on physics nobody verified. The two are complementary, and we will test them in combination. Where the physics is irrelevant, as in free-space reaching, $\Gamma$ is near zero and generated assets count at full value. Near a latch or a stiff hinge, where the guessed physics dominates the outcome, their influence is discounted.
Finally, we validate the penalty itself. ArtVIP assets come with measured joint dynamics, so we can generate language-model physics ranges for them too and check whether a high $\Gamma$ actually predicts a large error between the guessed and the measured dynamics. If it does not, the mechanism is not doing what we claim, whatever the downstream scores say.
This work differs from its nearest neighbors in specific ways. GTA adds diversity by denoising trajectories with a learned diffusion model inside one fixed environment. We add new objects and obtain counterfactuals from a physics engine, not a learned model. Statistical Guarantees for Offline Domain Randomization fits simulator parameters to real logs of a known system. We have no real logs for generated objects and use the unresolved parameter spread as a penalty rather than trying to resolve it. SR² resets a fixed handcrafted simulator to states found in real data. We generate the simulator's assets and never assume a matching real log.
Baselines
The naive reference point is offline RL (IQL and CQL) plus behavioral cloning, trained on a small curated set of ArtVIP assets from the training categories. This is the "hire artists, collect data" status quo. It uses the same one million transitions and the same behavior mixture, and it is evaluated on held-out assets. Every other method must beat it at equal transition count, or the generated diversity is not worth having.
The strongest simple competitor is behavioral cloning on the full generated corpus. The BC result in The Generalization Gap in Offline Reinforcement Learning makes this the default a skeptic expects to win. It receives the same data, the same vision encoder and the same tuning budget, measured in hyperparameter trials per method. We also run a BC variant filtered to successful trajectories, because filtering is how BC would realistically exploit mixed-quality data. The file reports no absolute Procgen or WebShop numbers for BC, so no reported number can anchor this comparison.
The strongest existing augmentation method is GTA applied to the curated dataset. It is reported to raise TD3+BC from 76.9 to 84.6 average normalized score on D4RL locomotion (GTA paper). That figure is from single-environment locomotion, so it motivates the baseline but is not a target on our benchmark. On articulated generalization, no reported result was found on Lacuna for GTA. We run it with its published return-guidance settings on the same curated data, followed by IQL.
For real-world transfer, the reference number is ArtVIP's action-chunking transformer on real drawer pulling. It is reported at a 39% zero-shot success rate when trained only in ArtVIP simulation, and 81% when mixed with real data (ArtVIP). Beating 39% zero-shot with a policy trained mainly on generated assets would be meaningful. Matching it with far less curated content would also count.
Two ablations isolate the key ingredient. The first uses physics-resampled replay as augmentation only, with $\beta = 0$. It separates the benefit of the extra counterfactual outcomes from the benefit of the penalty. The second randomizes physics at data-collection time with the same total transition count and no penalty. It separates "seeing varied physics" from "knowing which transitions depend on unverified physics."
Why not just…?
Why not just use behavioral cloning on the diverse corpus? On expert data in discrete games, BC won (The Generalization Gap in Offline Reinforcement Learning), and it may win again here. That would be a real finding, and we will report it. Our datasets are deliberately of mixed quality, however, and BC imitates failed attempts along with good ones. The experiment settles the question by sweeping the expert fraction of the data. If BC matches the penalized offline RL at every expert fraction, the method adds nothing and the correct conclusion is "scale assets, then clone."
Why not just randomize physics uniformly when collecting data? This is the most serious threat, because it is nearly free. Statistical Guarantees for Offline Domain Randomization gives looser transfer bounds for uniform randomization than for data-informed randomization, but that argument assumes real logs to fit against, which generated assets lack. Uniform randomization exposes the agent to varied physics but never tells the value function which outcomes to distrust. The collection-time randomization ablation measures exactly this. If it matches physics-resampled replay, the penalty is unnecessary, and we would drop it and report the diversity–fidelity exchange rate alone.
Why not just run online RL in the generated simulators, as RoboGen does? With a simulator available, online RL is legitimate, and we include PPO trained on the same assets under a matched simulator-step budget as a reference. The offline regime matters because large corpora gathered by planners and teleoperation are collected once and reused across many algorithms, and because the eventual target mixes in real logs that cannot be re-collected. We treat online PPO as an upper reference, not a competitor we must beat.
Why not just pretrain on synthetic noise? Pre-training with Synthetic Data Helps Offline Reinforcement Learning reports roughly 10% average gains for CQL after minutes of pretraining on random dynamics. Its authors attribute these gains to smoother optimization, not transferable semantics. The effect is orthogonal to ours, so we apply it to every method. If it closes the unseen-object gap by itself, the generated assets are not what matters.
Proposed Experiments
The primary benchmark is task success on held-out ArtVIP assets from subcategories absent from the curated training set, so that curated training has no unfair advantage over generated training. We also report success on held-out generated assets and, for drawer pulling, on the real robot. The decisive plots have two axes. The first is test success against the number of distinct training assets at a fixed one million transitions, for IQL, CQL, BC, and IQL with physics-resampled replay. This is the continuous-control analogue of the Procgen diversity sweep. The second is the exchange rate: the number of generated assets needed to match a given number of curated ones, with and without the penalty.
The central prediction is that unpenalized offline RL on generated assets plateaus or declines as low-fidelity assets are added, while the penalized version keeps improving. The gap should concentrate in latch and high-stiffness categories and be absent in free-motion ones. Three results would falsify the proposal. The first is $\Gamma$ failing to correlate with measured physics error on ArtVIP assets. The second is collection-time randomization matching the penalty. The third is BC matching everything at every expert fraction. We run five seeds per configuration with equal tuning budgets, and report 95% confidence intervals over held-out assets.
References
- The Generalization Gap in Offline Reinforcement Learning
- ArtVIP: Articulated Digital Assets of Visual Realism, Modular Interaction, and Physical Fidelity for Robot Learning
- RoboGen: Towards Unleashing Infinite Data for Automated Robot Learning via Generative Simulation
- CasTex: Cascaded Text-to-Texture Synthesis via Explicit Texture Maps and Physically-Based Shading
- Exploiting Action Impact Regularity and Exogenous State Variables for Offline Reinforcement Learning
- Is Pessimism Provably Efficient for Offline RL?
- GTA: Generative Trajectory Augmentation with Guidance for Offline Reinforcement Learning
- Statistical Guarantees for Offline Domain Randomization
- State Revisit and Re-explore: Bridging Sim-to-Real Gaps in Offline-and-Online Reinforcement Learning with An Imperfect Simulator
- Pre-training with Synthetic Data Helps Offline Reinforcement Learning