A robot policy learned by copying human demonstrations, called behavior cloning, can never do much better than the demonstrations it copied. A planner that searches through thousands of imagined futures in a simulator can do much better, but only if the simulator behaves like the real robot, and it never quite does. One line of work corrects a simulator with a small amount of real data. It learns an "action transformer" that rewrites each commanded action so that the simulator's next state matches what the real robot would do. So far that corrected simulator has only been used to train reactive policies by trial and error. We propose to use it as the forward model inside a search-based planner, distill the planner into a fast policy, and use that policy to widen the set of actions a real-robot self-improvement loop can consider.
The central difficulty, and the reason this is not a simple combination, is that planners are adversaries of their models. An optimizer that scores thousands of action sequences will find exactly the sequences where the corrected simulator is still wrong and report them as excellent. A correction fitted to demonstration or policy data is accurate where that data lives, not where the planner goes. We therefore propose planner-aware grounding. Real transitions used to refit the action transformer are collected along the trajectories the planner and its distilled student actually choose. The planner is penalized for entering regions where an ensemble of action transformers disagrees. The lookahead is cut short with a value function trained on real outcomes, so that the planner does not have to trust long rollouts of the corrected simulator. The claim is that this loop exceeds the demonstration ceiling with far fewer real episodes than learning a dynamics model from real data alone, and more reliably than either training a policy directly in the corrected simulator or planning in an uncorrected one.
Core Claim
A grounded simulator is a much better planning model than a policy-training model, provided the grounding data follows the planner. The expected outcome is a distilled policy whose real-world return rises above the behavior-cloning ceiling within tens of real episodes and approaches a planner given the true "real" dynamics. The ablation that separates this from prior work is where the grounding data comes from. If grounding on demonstration or behavior-cloning rollouts is as good as grounding on planner-visited states, the core claim fails.
Where This Sits in Prior Work
Three works supply the planning-to-policy machinery, and all three assume a perfect simulator. Deep Learning for Real-Time Atari Game Play Using Offline Monte-Carlo Tree Search Planning distills offline tree search into a pixel-based network. Its interleaved variant, which retrains on states the student itself visits (the DAgger scheme), was needed to prevent compounding errors. Thinking Fast and Slow with Deep Learning and Tree Search closes the loop: the apprentice network guides the search, and the search in turn produces the apprentice's training targets. Extracting Strong Policies for Robotics Tasks from Zero-Order Trajectory Optimizers, known as APEX, does the same for continuous robot control. It uses a sampling-based trajectory optimizer (iCEM, an improved cross-entropy method), warm-starts it with the student, and adaptively weights how closely the student must follow it. APEX's authors name model error as the unaddressed frontier, since errors in the optimizer's model flow straight into the distilled policy. We reuse APEX's optimizer and distillation scheme unchanged and replace its ground-truth MuJoCo model with a corrected simulator.
The correction method comes from grounded action transformation. Grounded Action Transformation for Robot Learning in Simulation composes a real forward model with a simulated inverse model. Reinforced Grounded Action Transformation for Sim-to-Real Transfer, called RGAT, instead trains the transformer by reinforcement learning to match real transitions, because the composed models accumulate errors when policies are deep networks. That result constrains our design: we build on RGAT, not classical GAT. RGAT uses the corrected simulator only to train a model-free policy, fits the correction on the policy's own data, and never exposes it to an optimizer. That combination is exactly the setting in which exploitation of residual model error has not been tested.
The ceiling side is motivated by Beyond Imitation: Self-Improving Robot Policies via Off-Policy Q-Planning. That work keeps a behavior-cloning policy frozen and trains a separate critic that reranks candidate action chunks sampled from it. The critic is a Q-function, a learned estimate of the future reward of taking an action in a state. The approach lifts real bimanual success substantially, but its authors state that it cannot promote actions the frozen prior never proposes. Our distilled planner student is designed as a second proposal source that fills this gap. Digital-twin pipelines take the neighboring route of widening exploration with simulation. TwinRL-VLA: Digital Twin-Driven Reinforcement Learning for Real-World Robotic Manipulation runs reinforcement learning in a scanned twin, and X-Sim: Cross-Embodiment Learning via Real-to-Sim-to-Real trains in a reconstructed scene. Both leave physical parameters ungrounded, and TwinRL-VLA reports that twin failures do not always match real ones. That observation is the symptom our planner-targeted grounding is meant to fix. Finally, Reflective Planning: Vision-Language Models for Multi-Stage Long-Horizon Robotic Manipulation shows the alternative of a learned visual forward model. It is limited to one-step lookahead by compounding hallucination, which is one reason we keep a physics engine as the backbone of the model.
Research Program
The system has four parts that run in rounds. The first is a physics simulator with an ensemble of RGAT-style action transformers, each trained on a different bootstrap sample of the real transitions gathered so far. The second is an iCEM planner that scores an $H$-step action sequence $a_{1:H}$ from a state $s$ by rolling it out through a randomly chosen ensemble member:
$$J(a_{1:H}) = \sum_{t=1}^{H} \big[ r(\hat s_t, a_t) - \beta\, \sigma_g(\hat s_t, a_t) \big] + V_{\text{real}}(\hat s_H).$$
Here $\hat s_t$ is the predicted state after $t$ steps, $r$ is the task reward, and $\sigma_g$ is the spread of next-state predictions across the ensemble of transformers, which is high wherever the correction has not been fitted. The weight $\beta$ sets how strongly the planner avoids such regions. $V_{\text{real}}$ is a critic trained off-policy only on real transitions, and it lets $H$ stay short. The third part is a student policy distilled from the planner with APEX's adaptive guidance and DAgger-style relabeling. The fourth is a real-robot deployment step. Each round the student runs on the real system for a fixed small number of episodes, perturbed toward the planner's preferred actions, and every real transition is added both to the grounding dataset and to the critic's replay buffer.
At deployment, candidate action chunks come from two sources, the original behavior-cloning prior and the distilled student. They are reranked by the real critic using the Q-weighted average from Q-Planning, which that paper found more stable than picking the single best candidate. The analysis component measures, round by round, the transformer's one-step and $H$-step prediction error on three distributions: demonstration states, behavior-cloning rollout states, and planner-visited states. This tests the mechanism directly. The prediction is that error on planner-visited states is initially largest and falls fastest under planner-aware grounding.
Baselines
The naive reference point is the behavior-cloning policy alone. We train Diffusion Policy, from Diffusion Policy: Visuomotor Policy Learning via Action Diffusion, on the same demonstrations, because it avoids the averaging failure of regression-based cloning and reports 95% on Push-T against 20% for implicit behavior cloning. Its real-system success defines the ceiling to beat. A second reference is APEX planning in the uncorrected nominal simulator and distilled as-is, which measures the raw gap between simulation and the target dynamics. The oracle upper bound is APEX run with the true target dynamics as its model.
The strongest competitor on the ceiling question is Q-Planning. It is run with the same frozen prior, the same critic architecture scaled to the task, and the same number of real episodes as our method, with its self-improvement loop given all of them. The file reports its gains as a rise in success from 93% to 99% on LIBERO-10, 83.8% to 91.4% on RoboTwin, and 40% to 90% on real stack-cups. Beating it means a higher final success rate at equal real-episode budget, and specifically success on tasks where the prior's initial rate is near zero, which is where its support limit binds.
The strongest competitor on the grounding question is RGAT as published. The action transformer is fitted on the policy's own data, and a model-free policy is then trained in the corrected simulator with the same simulation compute we give the planner. No numeric RGAT result on these tasks is reported in the file (no reported result found on Lacuna for RGAT on iMuJoCo or manipulation). Beating it means higher real return at equal real transitions.
Two ablations isolate the key ingredient. The first swaps where grounding data comes from: behavior-cloning rollouts only versus planner and student rollouts, with the real-sample count identical. The second removes the disagreement penalty and the real-data critic in turn. All methods share the demonstrations, the real-episode budget, and simulation compute. Each gets the same hyperparameter search budget, tuned on a held-out set of target dynamics variants.
Why not just…?
Why not just learn a dynamics model from the real data and plan with it, skipping the simulator? This is the most serious threat. With enough real data, a learned ensemble model plus iCEM could match us, and the file contains no result that rules this out. Our bet is that the physics engine provides a prior that extrapolates, so the action transformer only has to learn a small correction. That bet should matter most at small real budgets. We settle it by giving a learned-ensemble iCEM planner exactly the same real transitions and sweeping the budget. If it matches us at every budget, the simulator adds nothing and the proposal reduces to a model-based reinforcement learning result.
Why not just use RGAT and train a policy in the corrected simulator, as its authors did? RGAT fits its correction where its own policy goes. A policy trained by trial and error inside the corrected simulator can also exploit residual errors, but more slowly and less directly than an optimizer that searches thousands of sequences per step. The head-to-head comparison at equal real data, together with the per-distribution prediction-error analysis, shows whether search in the corrected simulator buys anything beyond its use for policy training.
Why not just randomize the simulator's physics and plan robustly? Randomizing parameters such as mass and friction covers the target in expectation but makes the planner optimize for the worst plausible world. The file notes this tends to produce conservative behavior on precise contact tasks, and it reports that SURREAL: Open-Source Reinforcement Learning Framework and Robot Manipulation Benchmark left Bin Picking and Nut-and-peg Assembly unsolved even with 64 actors. We include APEX over a domain-randomized simulator, with ranges chosen to include every target variant, as a baseline. If it matches us on the locomotion variants but loses on the contact-rich manipulation variants, that locates where grounding is worth its cost.
Why not just collect more demonstrations or fine-tune on the successful rollouts? Q-Planning reports that filtered fine-tuning on successes stalled at 55% and 30% on its real tasks, while the critic-based method reached 90% and 80%. Offline Q-learning on Diverse Multi-Task Data Both Scales And Generalizes found that a supervised sequence model failed to exceed suboptimal data where value learning reached 77.8% human-normalized. We still run filtered fine-tuning at our real budget as a cheap reference.
Proposed Experiments
Following RGAT's protocol, the main study uses sim-to-sim surrogates: nominal MuJoCo plays the simulator, and a perturbed MuJoCo plays "reality." For locomotion, targets are drawn from the 154 mass, friction, limb-length, and joint-range variants of mpatacchiola/imujoco, introduced in Comparing the Efficacy of Fine-Tuning and Meta-Learning for Few-Shot Policy Imitation. Their 100 SAC rollouts per variant are subsampled and truncated to create a deliberately suboptimal demonstration set, so that a ceiling exists. That paper's finding, that plain fine-tuning matches meta-learning from 10 trajectories on, justifies using fine-tuning rather than meta-learning for the behavior-cloning baseline's adaptation. For manipulation, we use APEX's Fetch Pick&Place and ShadowHand Door and Robomimic Can. On these we add unmodeled friction changes, actuator delay, and mass changes, with demonstrations limited so that Diffusion Policy plateaus well below the oracle. APEX reports solving these tasks with high success in the true simulator but gives no number in the file (no reported result found on Lacuna for APEX success rates), so we rerun it as our oracle.
The primary metric is real-system success or return as a function of real episodes consumed. Secondary metrics are the fraction of the gap between behavior cloning and the oracle planner that is closed, transformer prediction error on each state distribution, and the student's inference latency. The decisive checks, fixed in advance, are these. At a budget of 50 real episodes, our method should close at least half of the gap between behavior cloning and the oracle on the manipulation variants. The planner-visited grounding ablation should beat behavior-cloning-rollout grounding by more than seed variance. Our method should beat the learned-ensemble planner at the lowest budgets. Failing the second check falsifies the mechanism; failing the third means the simulator prior is not earning its place. A final, smaller physical study, run only if the surrogate results hold, repeats the comparison with Q-Planning on a single contact-rich insertion task, since RGAT itself was never validated on hardware.
References
- Deep Learning for Real-Time Atari Game Play Using Offline Monte-Carlo Tree Search Planning
- Thinking Fast and Slow with Deep Learning and Tree Search
- Extracting Strong Policies for Robotics Tasks from Zero-Order Trajectory Optimizers
- Grounded Action Transformation for Robot Learning in Simulation
- Reinforced Grounded Action Transformation for Sim-to-Real Transfer
- Beyond Imitation: Self-Improving Robot Policies via Off-Policy Q-Planning
- TwinRL-VLA: Digital Twin-Driven Reinforcement Learning for Real-World Robotic Manipulation
- X-Sim: Cross-Embodiment Learning via Real-to-Sim-to-Real
- Reflective Planning: Vision-Language Models for Multi-Stage Long-Horizon Robotic Manipulation
- Diffusion Policy: Visuomotor Policy Learning via Action Diffusion
- SURREAL: Open-Source Reinforcement Learning Framework and Robot Manipulation Benchmark
- Offline Q-learning on Diverse Multi-Task Data Both Scales And Generalizes
- Comparing the Efficacy of Fine-Tuning and Meta-Learning for Few-Shot Policy Imitation