Robots trained in simulation are usually made robust by domain randomization: masses, friction, damping, and payloads are resampled every episode so the policy cannot overfit to one simulator. This makes policies robust on average. It does not stop them from swinging an arm too fast or slamming into contact during training or on hardware. Safe reinforcement learning methods try to stop that. They either enforce constraints only once training has converged, or they place a model-based shield in front of the policy, and the shield's proof breaks as soon as the physics parameters change. We propose a controller whose safety argument does not depend on the randomized parameters at all. The controller has two parts. The first is a stabilizing core that pulls the robot toward a target in a way that can only remove energy. The second is an unrestricted neural "residual" policy that adds to the core's command, and the energy it can inject is metered by a running energy budget measured from the robot's own sensors.
The key observation is that for mechanical systems, the map from motor torque to joint velocity is passive, meaning the system can only store or dissipate the energy pushed into it. This holds for every mass, inertia, friction coefficient, and damping value a randomizer might sample. A controller that is itself passive, plus a residual whose injected power is counted and capped, therefore bounds the robot's total mechanical energy, and through it the joint speeds and impact forces, in every sampled simulator and on the real robot, with no model of the parameters. Because safety comes from the architecture rather than from a learned constraint estimate, the residual can be trained with any off-policy algorithm using replay across randomized domains. That is where the sample-efficiency gain should come from.
Core Claim
The controller is
$$u_t = -\nabla_q \Phi(q_t - q^*;\theta) - D(q_t,\dot q_t;\theta)\,\dot q_t + \beta_t\,\pi_\psi(o_t),$$
where $q$ and $\dot q$ are joint positions and velocities, $q^*$ is a target configuration, $\Phi$ is a strictly convex neural potential, $D$ is a learned positive semi-definite damping matrix, $\pi_\psi$ is an unconstrained residual policy acting on observations $o_t$, and $\beta_t \in [0,1]$ is a smooth gate. The gate is driven by an energy tank $T_t$. The tank is drained by the measured residual power $\beta_t \pi_\psi(o_t)^\top \dot q_t$ and refilled by a fixed fraction of the power dissipated through $D$. It is capped at $T_{\max}$, and $\beta_t = \mathrm{clip}(T_t/\epsilon, 0, 1)$ fades the residual out continuously as the tank empties.
The total energy $H$ is the kinetic energy plus $\Phi$. In continuous time, $H$ can then never exceed its initial value plus $T_{\max}$. The guarantee depends on the randomized parameters $\xi$ in exactly one place: the speed bound $\|\dot q\| \le \sqrt{2(H_0 + T_{\max})/\lambda_{\min}(M(\xi))}$. Here $M(\xi)$ is the mass matrix and $\lambda_{\min}$ its smallest eigenvalue, so the bound is computed from the lightest configuration in the randomization range. The tank needs only measured torques and velocities, so the certificate holds for every simulator in the distribution and for the real robot without identifying any parameter.
The claim has deliberate limits. It certifies energy-type constraints: speed limits, bounded impacts and contact forces, and no runaway motion. It does not certify geometric constraints such as obstacle avoidance. It also covers only the parameters that preserve passivity. Actuation latency and discrete-time sampling can inject energy, and we treat them as the central risk rather than hiding them.
Relation to Prior Work
The stabilizing core is taken from stability-by-design work. Structured Neural-PI Control writes the controller as the gradient of a strictly convex network and proves stability through passivity that does not depend on the equilibrium chosen. It was demonstrated on power grids and vehicle platoons, with no exploration and no randomization. We reuse its convex-gradient parametrization as the core, and add the residual and tank so the controller can learn non-monotone behaviour.
Monotone neural barrier certificates and input-convex Lyapunov controllers in Neural Lyapunov Differentiable Predictive Control (NLDPC) also obtain guarantees from structure. NLDPC, however, explicitly loses its guarantee under plant mismatch. Our argument avoids this because passivity is a property of the whole plant class rather than of a nominal model.
Contraction-based modular RL is the closest stability-plus-exploration work. It constrains policy gradients so that trajectories contract. It needs local linear models and online eigendecompositions, and it never randomizes the dynamics. We need neither a model nor online linear algebra.
Prior safe sim-to-real work shows the gap but adopts designs we avoid. Sim-to-Lab-to-Real switches between an unconstrained task policy and a backup policy using a learned reachability critic. REVEL projects onto verified symbolic shields, at roughly 87% of its training time, under a fixed worst-case model. LyEvO certifies a stability region statistically using evolutionary search, at over 31 hours on up to 1,000 CPU cores, and reports actuator jitter. UARL gates deployment on critic-ensemble disagreement, which can be falsely confident. Provably Safe Sim-to-Real Transfer mixes a known safe policy with an exploratory one, but only in tabular settings.
Our proposal differs from all five in the same way. The safety mechanism is continuous, which removes switching and chattering. It is model-free with respect to the randomized parameters. And it costs a few multiply-adds per control step.
On the learning side, two papers constrain the design. FOCOPS and the other primal-dual methods rely on on-policy sampling, which makes them expensive to combine with randomization. Peng et al. show that randomized manipulation needs off-policy replay and hindsight relabeling to be practical. Decoupling safety from the learner lets us use the latter.
Research Program
We will build the controller in a joint-torque interface for MuJoCo robots. The core is a strictly convex network with softplus-type activations and a damping network $D = LL^\top + \delta I$, where $L$ is a learned matrix and $\delta$ a small positive constant that keeps $D$ positive definite. We train three versions of the residual. The first is soft actor-critic (an off-policy learner that maximizes reward plus action entropy) with replay pooled across randomized episodes. The second adds hindsight relabeling for goal tasks. The third conditions the residual on a recurrent context encoder, following Peng et al. and context-aware transfer. The core parameters $\theta$ are trained jointly through the same actor loss. Gradients flow through $\beta_t$ but never alter the tank accounting.
The theoretical work has two parts. First, we will prove the continuous-time energy bound and the speed and force bounds for the full class of rigid-body systems with dissipative contact. Second, we will derive a discrete-time version. The tank will charge the residual for the worst-case work it could do over one control interval, given a measured bound on joint acceleration. A small damping margin will absorb the energy that a known maximum latency $\tau$ could inject.
That second bound predicts exactly when the guarantee should fail. When the randomized latency exceeds the $\tau$ that the margin was sized for, violations should appear. We will test that prediction directly.
Baselines
The naive reference is soft actor-critic or proximal policy optimization (PPO) trained with the same domain randomization and the same raw torque action space. The constraint enters only as a scalar reward penalty with a fixed weight. This is the setting that Quanyi Li et al. show to be brittle in both directions: too small a penalty accepts failures, too large a penalty freezes the agent.
The strongest constrained-RL competitors are PPO-Lagrangian and the PID-damped Lagrangian variant (CPPO-PID) from Safety Gymnasium, together with FOCOPS. They are the standard methods for velocity-constrained locomotion and the closest in spirit, since they enforce the same speed constraint through optimization. The reported reference points are a 98% reduction in safety violations relative to PPO on Safety-Gymnasium velocity tasks, at the cost of a 45% drop in total reward (Safety Gymnasium). For FOCOPS, the file reports wins over CPO and PPO-Lagrangian on 5 of 6 MuJoCo speed-limit tasks. The only exact score it gives is a HumanoidCircle reward of 1106.1 at cost ≤ 50, which is a position constraint outside our scope. No per-task speed-limit score was found on Lacuna for FOCOPS, so we will rerun it.
For the switching-shield family, we implement a backup-policy shield gated by a learned safety critic in the style of Sim-to-Lab-to-Real. For manipulation we compare against the recurrent randomized policy of Peng et al., which reports 0.89 ± 0.06 real success on Fetch puck pushing, and the policy-loss context encoder of Iannotta et al., which reports 63% real success versus 32% for context-agnostic randomization on Franka box pushing.
All methods receive the same randomization distribution, the same environment-step budget, the same network sizes where applicable, and the same hyperparameter search budget, set at an equal number of tuning trials. All methods are evaluated on the same held-out parameter sets.
Beating the strongest baselines means two things together. First, zero constraint violations during training across all seeds, where the Lagrangian methods incur violations early. Second, final constrained return at least equal to PPO-Lagrangian's, reached with fewer environment steps.
Two ablations isolate the key ingredients. To test whether the learned core matters, we replace the convex core with a fixed proportional-derivative (PD) controller, a hand-tuned spring-damper that pulls toward the target. To test whether the tank is the right safety mechanism, we keep the core and replace the tank either with a magnitude clip on the residual or with a hard on/off switch.
Why not just…?
Why not just add a Lagrangian cost to PPO or SAC with randomization? Lagrangian methods keep the constraint only after the multiplier has settled. Safety Gymnasium documents oscillation between safe-and-slow and unsafe-and-fast regimes, and a 45% reward cost for a 98% cut in violations. Under randomization the multiplier also has to track a moving target. Our training-time violation curves will show directly whether this alternative violates constraints early, where ours should not.
Why not just randomize more widely and trust robustness to emerge? Wide randomization is known to produce sluggish policies (UARL) and offers no bound outside the training range. Unconstrained networks diverged under out-of-range inputs in Deep Input-Output Stable Dynamics, and deep RL baselines showed up to 54% violation rates in LyEvO. We test this alternative explicitly with masses and frictions outside the training range.
Why not just put a residual on top of a hand-tuned PD controller and clip its output? This is the most serious threat. A PD controller is already passive, and residual RL on a classical controller is known to save samples: Yamada et al. report roughly 800 trials reduced to 50. Clipping, however, bounds force rather than energy. A clipped residual can still pump energy into a resonant mode one step at a time.
We expect the two halves of our design to contribute different things. The tank should supply most of the safety. The learned convex core should supply the larger basin of attraction and better transients; Neural-PI reports over 30% less wasted energy than optimally tuned linear PI. If "PD + tank" matches our full method on both safety and return, we will report that the contribution is the tank, not the learned core.
Why not just use a learned shield or a verified stability region? Switching shields cause chattering at the handoff (Sim-to-Lab-to-Real, Yamada et al.). Verified approaches are expensive: about 87% of training time in REVEL and 31 hours on up to 1,000 cores in LyEvO. Both are also tied to a nominal model. We will measure high-frequency torque content, as a proxy for chattering, alongside wall-clock compute.
Proposed Experiments
Velocity-constrained locomotion. We use the velocity tasks in Safety-Gymnasium and the MuJoCo speed-limit suite from FOCOPS (Ant, HalfCheetah, Hopper, Walker2d, Humanoid, Swimmer), each wrapped with randomization of link masses (±30%), joint damping, and ground friction. Metrics are cumulative constraint violations over training, final return under the speed limit, and environment steps needed to reach 90% of the best baseline's final return. Here the tank budget $T_{\max}$ is set from the speed limit through the kinetic-energy bound, using the lightest mass in the randomization range.
Manipulation transfer. We use simulated Fetch puck pushing with the randomization ranges of Peng et al., and Franka box pushing over masses of 283–824 g following Iannotta et al.. Metrics are success rate, peak contact force, and peak joint speed on held-out and out-of-range parameter sets. Hardware runs on a Franka arm, if available, use the same protocol.
Contact-force constraint. We use the peg-maze task from contraction-based RL, with its reported 99.93% rate of keeping contact below 1 N for C-TD3 versus 65.88% for HIRO, and add mass and friction randomization. This tests whether the energy bound keeps contact force low without contraction metrics.
Decisive checks. Three outcomes would settle the claim. First, the full method must show zero speed-limit violations during training across all seeds and all held-out parameters within the passivity-preserving class. A single violation there indicates a bug or a broken assumption. Second, a latency sweep that crosses the designed $\tau$ should show violations appearing near the predicted threshold. If they appear well below it, the discrete-time bound is wrong. If they never appear, the bound is loose and we will tighten it. Third, the PD+tank and core+clip ablations will show whether the safety comes from energy metering and whether the learning benefit comes from the learned core. The proposal stands only if both the full method's safety and its sample-efficiency advantage over Lagrangian training hold up against these ablations.
References
- Structured Neural-PI Control with End-to-End Stability and Output Tracking Guarantees
- Monotone Neural Control Barrier Certificates
- Neural Lyapunov Differentiable Predictive Control
- Stable modular control via contraction theory for reinforcement learning
- Sim-to-Lab-to-Real: Safe RL with Shielding and Generalization Guarantees
- Neurosymbolic Reinforcement Learning with Formally Verified Exploration
- LyEvO: Lyapunov-Guided Evolutionary Optimization for Safe and Robust Sim-to-Real Policy Learning
- Safe Domain Randomization via Uncertainty-Aware Out-of-Distribution Detection and Policy Adaptation
- Provably Safe Sim-to-Real Transfer
- First Order Constrained Optimization in Policy Space
- Sim-to-Real Transfer of Robotic Control with Dynamics Randomization
- Can context bridge the reality gap? Sim-to-real transfer of context-aware policies
- MetaDrive: Composing Diverse Driving Scenarios for Generalizable Reinforcement Learning
- Safety Gymnasium: A Unified Safe Reinforcement Learning Benchmark
- Learning Deep Input-Output Stable Dynamics
- Hybrid Reinforcement Learning and Its Application to Biped Robot Control