When a robot policy is trained in a physics simulator and then deployed on hardware, it usually performs worse, because the simulator's friction, delays and contacts differ from reality. One practical fix, called grounded simulation learning, uses a small number of real trajectories to learn an "action transformer." This is a small network that rewrites each action before the simulator executes it, so that simulated transitions look like real ones. The policy is then retrained in the corrected simulator, and the two steps alternate. These methods share a blind spot. The transformer is fit on states the current policy happened to visit. Each retrained policy then tends to move into regions where the correction is worst, because simulator errors there are easy to exploit. The alternation is a heuristic with no account of this feedback and no convergence analysis.
We propose to treat grounding as what it is: a two-level (bilevel) optimization problem. The outer variable is the action transformer. The inner problem is policy optimization in the grounded simulator. The outer objective measures how wrong the simulator is on the states the resulting policy will visit. Recent theory makes this tractable. Under the Polyak-Łojasiewicz (PL) condition, a property weaker than convexity, an objective's gradient stays large wherever the objective is far from optimal. Bilevel problems whose inner level satisfies PL admit first-order algorithms with convergence guarantees, and entropy-regularized policy optimization satisfies a generalized PL condition. The result is a grounding update that anticipates how the policy will respond. We will test whether it reduces the real data needed and stops the oscillation that plain alternation suffers.
Core Claim
Existing grounding methods omit one term from the update of the action transformer: the change in the policy's visited-state distribution caused by changing the transformer. Both Grounded Action Transformation (GAT) and Reinforced GAT (RGAT) leave it out. GAT composes a real forward model with a simulator inverse model. RGAT trains the transformer end-to-end with reinforcement learning against transition errors. Both treat the data distribution as fixed when updating the transformer.
We claim three things. First, including the missing term changes the transformer's behavior in a specific way: it is pushed to be accurate where the next policy will go, not where the last one went. Second, the term can be estimated without second derivatives, using a penalty reformulation that is valid when the inner policy problem is PL. Third, the resulting single-loop algorithm reaches a given transfer performance with fewer real trajectories and shows monotone rather than oscillating real-world return across grounding rounds.
The claim is falsifiable. If the response term does nothing beyond what simple data aggregation or a reward penalty already achieves, the proposal fails.
Formulation
Let $\phi$ parameterize the action transformer $g_\phi(s,a)$ and $\theta$ the policy $\pi_\theta$. Let $d_\phi^{\theta}$ be the state-action distribution of $\pi_\theta$ in the simulator grounded by $g_\phi$. Let $\Delta(s,a;\phi)$ measure the discrepancy between the grounded simulator's next state and an ensemble of forward models fit to all real data, plus the ensemble's disagreement, so that unvisited regions count as uncertain. The grounding problem is
$$\min_\phi \; F(\phi) = \mathbb{E}_{(s,a)\sim d_\phi^{\theta^*(\phi)}}\big[\Delta(s,a;\phi)\big] \quad \text{s.t.} \quad \theta^*(\phi)\in\arg\max_\theta J_\phi^\tau(\theta),$$
where $J_\phi^\tau$ is the entropy-regularized simulated return with temperature $\tau$.
We follow the Hessian-free bilevel actor-critic, which supplies our method. That paper proves the inner problem satisfies a generalized PL condition, but only for reward-parameterized outer variables and only on GridWorld and language-model tasks. Following its approach, we replace the constraint with a penalty on the inner optimality gap:
$$\min_{\phi,\theta}\; F(\phi,\theta) + \lambda_t\big(V^{*,\tau}_\phi - J^\tau_\phi(\theta)\big).$$
Here $V^{*,\tau}_\phi$ is tracked by a second reference actor-critic. The penalty weight $\lambda_t$ grows and the temperature $\tau$ decays over training. PL is what makes this penalty a faithful surrogate: the optimality gap is bounded by the squared gradient norm, so driving the gradient to zero drives the policy to the inner solution set. The gradient of $F$ with respect to $\theta$ through $d_\phi^\theta$ is an ordinary policy gradient with reward $-\Delta$. The new element is that the outer variable here is a dynamics-modifying transformer rather than a reward, so $\phi$ enters both the transition kernel and the outer loss.
Research Program
The work has three parts.
Analysis. We will extend the convergence argument of the Hessian-free actor-critic to the case where $\phi$ changes transitions rather than rewards. Smoothness of the occupancy measure in $\phi$ then has to be assumed or enforced. We will enforce it by parameterizing $g_\phi$ as a residual on the identity with a Lipschitz bound, borrowing the certified construction from BiLipNet/PLNet. That paper builds networks with certified Lipschitz bounds and PL guarantees but never applies them to control. Here we use only the Lipschitz part, as a regularity device.
Theory caveats. We will be explicit about two limits. First, GALET proves that a second-order term is unavoidable for exact stationarity under non-strongly-convex PL inner problems. Penalty methods therefore converge to penalty-stationary points, and we will measure how far that sits from the true bilevel solution on small problems where it can be computed. Second, GALET and HJFBiO, which approximates Hessian-vector products by finite differences, are deterministic analyses. We will check whether finite-difference variants survive policy-gradient noise.
Empirical PL checks. We will measure whether PL plausibly holds for the neural policies used. We estimate the ratio $\|\nabla J\|^2 / (J^\star - J)$ along training trajectories, with $J^\star$ proxied by long-run training from many seeds. A ratio that collapses toward zero in some environments would tell us where the theory does not apply.
Baselines
Do-nothing reference. A policy trained in the ungrounded simulator and deployed directly, together with a policy trained on the target environment as an upper reference. RGAT reports its results relative to this upper reference.
Domain randomization. We will run domain randomization over the same physics ranges using the RRLS implementation. The RRLS paper reports it as a strong average-case baseline across uncertainty sets, though no return values are given in the file. It uses no target data, so we compare at matched simulation steps.
RGAT. The strongest existing grounding method, and the most direct competitor because it shares our action-transformer parameterization. It reports matching a directly trained policy on modified InvertedPendulum, Hopper and HalfCheetah. However, no reported result was found on Lacuna for RGAT's numerical returns, since they appear only in figures. We will reimplement it with the same 64-by-64 policy, the same transformer architecture and the same target-trajectory budget, and tune it with the same number of hyperparameter trials as our method.
GAT. The classical grounding baseline. On real NAO hardware it reports walking at 27.97 cm/s versus 19.52 cm/s for the hand-tuned engine, using 65 hardware trajectories.
Parameter search. Auto-tuned sim-to-real represents simulator-parameter identification. It reports 90% and 40% real success versus 0% for domain randomization on two xArm tasks.
Ablations. Two ablations isolate the key ingredient. The first is a "myopic bilevel" version: identical code with the policy-response term in the outer gradient set to zero. The second is "aggregated RGAT," which fits the transformer on all real data collected so far.
What counts as success. Beating RGAT means reaching 90% of the directly-trained policy's target return with at least 30% fewer target trajectories across seeds, or lower round-to-round variance in target return at equal budget. The improvement must hold with confidence intervals over ten seeds.
Why not just…?
Why not just aggregate real data across rounds and refit, as a data-aggregation loop would? Aggregation fixes the transformer's coverage of states already visited. It does nothing for the regions a new policy will enter next, which is exactly where grounding errors get exploited. It is nonetheless the most serious threat, since with enough rounds it may close the gap. The aggregated-RGAT ablation settles this. If it matches our method at every target-data budget, the bilevel term is not worth its cost and we will report that.
Why not just add the model-uncertainty penalty to the policy's reward? This is close to what our $\theta$-update does at fixed $\lambda$, so it is a real competitor. The difference is that the reward-penalty version leaves the final policy permanently conservative. Our method instead decays the penalty's influence while shifting the burden onto the transformer. We expect the gap to show on nominal return, echoing the conservatism of worst-case methods documented in RRLS. We include this baseline with tuned penalty weight. If it ties our method, the contribution reduces to the analysis.
Why not just use domain randomization? It needs no real data. However, Auto-Tuned Sim-to-Real reports it achieving 0% on contact-sensitive hardware tasks where targeted identification succeeds. It is still included at matched simulation compute.
Why not use GALET's exact second-order update? It is the principled alternative, but it requires Hessian-vector products through policy gradients, which are noisy and costly. We run it only on InvertedPendulum as a reference for how much the penalty approximation loses.
Proposed Experiments
Main experiments. We follow RGAT's sim-to-"real" protocol. The source simulator is standard MuJoCo, and the target is the same task with modified physics. Tasks are InvertedPendulum, Hopper, HalfCheetah, and Walker2d from RRLS. The RRLS uncertainty sets over mass, friction and damping let us sample twenty target environments per task rather than one, so results are not tuned to a single shift. The primary metric is target-environment return, normalized by the directly-trained policy, plotted against the number of target trajectories, which range from 5 to 65 to mirror GAT's hardware budget. Secondary metrics are round-to-round variance of target return and transformer error on the final policy's visited states versus on earlier policies' states.
Decisive check. The mechanism predicts a specific signature. Under myopic grounding, the transformer error on each new policy's states should exceed its error on the previous policy's states. Under our method, that gap should shrink. If the gap does not shrink, the response term is not doing what we claim, regardless of return.
Secondary checks. We will report the measured gradient-dominance ratios per environment, and correlate them with where our method helps. If gains concentrate where PL appears to hold, that supports the analysis. If gains appear where it clearly fails, that undercuts it.
Hardware stretch. If simulated results are positive, a single xArm or NAO-style task would test whether the trajectory savings survive real sensing noise. We treat this as optional rather than as the core evidence.
References
- Grounded Action Transformation for Robot Learning in Simulation
- Reinforced Grounded Action Transformation for Sim-to-Real Transfer
- A Hessian-Free Actor-Critic Algorithm for Bi-Level Reinforcement Learning with Applications to LLM Fine-Tuning
- Monotone, Bi-Lipschitz, and Polyak-Łojasiewicz Networks
- An Alternating Optimization Method for Bilevel Problems under the Polyak-Łojasiewicz Condition
- Optimal Hessian/Jacobian-Free Nonconvex-PL Bilevel Optimization
- Rrls: Robust reinforcement learning suite
- Auto-Tuned Sim-to-Real Transfer