Engineers who train robots with reinforcement learning (RL) spend much of their time tuning penalty weights. They add terms for body tilt, joint torque and jitter to the task reward, then adjust the coefficients until the robot stops wobbling. The result works for one noise level and one motor limit. This note proposes replacing those weights with a single number that has physical units: a variance budget, the largest long-run average squared deviation of the state the designer will accept. The learner trains a second network, a "drift critic," which must certify that the expected energy of the state shrinks by a fixed fraction each step, up to a constant. Control theory shows that this kind of certificate bounds the long-run second moment of the state even when the noise is unbounded (Gaussian) and the motors saturate. That property is called mean-square boundedness. The budget fixes the constants in the certificate. A Lagrange multiplier then enforces it, so nobody has to hand-tune a weight.
The opportunity comes from a mismatch between two literatures. Stochastic control has shown that with bounded actuators and noise of unbounded support, convergence to a point is impossible. Only a finite variance envelope can be guaranteed, as shown in Stochastic receding horizon control with output feedback and bounded control inputs and Mean-square boundedness of stochastic networked control systems with bounded control inputs. The closest RL method, Lyapunov Actor-Critic in Reinforcement Learning for Control with Probabilistic Stability Guarantee, removes hand-tuned stability penalties by constraining a learned Lyapunov function to decrease in expectation. It targets asymptotic stability, however, and handles neither actuator limits nor unbounded noise. Under exactly those conditions, its decrease condition cannot be satisfied near the equilibrium.
Core Claim
The claim is that the right replacement for hand-tuned stability penalties is a geometric drift condition with an offset, not a pure decrease condition. Let $V_\phi(x)$ be a learned function of the state $x$, parameterized so that $V_\phi(x) \ge c\|x\|^2$ for a fixed constant $c>0$. One way to do this is $V_\phi(x) = c\|x\|^2 + f_\phi(x)^\top f_\phi(x)$, which mirrors the positive-definite critic of LAC. We require
$$\mathbb{E}\left[V_\phi(x_{t+1}) \mid x_t\right] \le (1-\alpha)\,V_\phi(x_t) + b,$$
where $\alpha\in(0,1)$ is a contraction rate and $b\ge 0$ is an offset that absorbs the noise the actuators cannot cancel. Iterating this expectation gives $\limsup_t \mathbb{E}[V_\phi(x_t)] \le b/\alpha$, and therefore
$$\limsup_{t\to\infty}\mathbb{E}\|x_t\|^2 \le \frac{b}{c\,\alpha}.$$
The designer states a budget $B$. The method sets $b = B c \alpha$, enforces the drift inequality on sampled transitions with an adaptive multiplier, and otherwise optimizes only the unshaped task reward. Setting $b=0$ recovers LAC's condition. Under persistent Gaussian noise with bounded inputs, that condition cannot be met in a neighborhood of the origin. We predict that LAC's multiplier diverges, or that its policy saturates permanently, in exactly the regimes where the offset version stays well-posed.
The proposal also yields a second, practical output. When the budget cannot be achieved with the available actuator authority, the multiplier grows without bound. By sweeping $B$, the method reports the smallest budget it can certify for a given actuator limit. This is a learned, empirical counterpart to the actuator-authority thresholds that Mean-square boundedness of stochastic networked control systems with bounded control inputs derives analytically for marginally stable linear systems.
Research Program
The algorithm is a maximum-entropy actor-critic, as in LAC, with three changes. First, the policy has a hard saturation (tanh scaled to the known limit $U_{\max}$). Second, the Lyapunov critic is replaced by the drift critic above. Third, the offset is tied to the budget. The drift critic is trained on replay transitions, and its expected next value is estimated from several sampled successor states when a simulator allows resampling.
There is an honest limitation here. Like LAC's data-based theorem, enforcing the inequality in expectation over the visitation distribution is weaker than enforcing it pointwise. The bound $b/(c\alpha)$ is therefore a design target that we validate by Monte Carlo, not a deployment proof. Where dynamics are linear and known, we check pointwise satisfaction directly against the learned $V_\phi$ on a grid.
The program has three stages. The first is analytic sanity. On linear systems with Gaussian noise and saturated inputs, the true minimal achievable variance can be bounded or approximated. There we compare the method's certified budget and empirical variance with stochastic MPC. One comparison point is the saturated, innovations-based design of Stochastic receding horizon control with output feedback and bounded control inputs. Another is the chance-constrained formulations that tighten constraints from the first two moments, in Robustifying Model Predictive Control of Uncertain Linear Systems with Chance Constraints and Stochastic Model Predictive Control for Linear Systems using Probabilistic Reachable Sets. This stage also probes the open problem of open-loop unstable plants, which the analytic proofs exclude. Here we expect the method to report infeasibility honestly, not to solve the problem.
The second stage is nonlinear benchmarks with injected white noise, where the comparison is against hand-shaped rewards. The third is a locomotion case: we remove tilt, torque and jitter penalties and keep only a forward-progress reward plus a variance budget on body attitude and height. We do this because Hierarchical Apprenticeship Learning with Application to Quadruped Locomotion and Learning Agile Locomotion via Adversarial Training identify that weight balancing as the bottleneck. The second paper reports the jitter that appears when no stability bound is enforced.
Baselines
The naive reference is the unshaped task reward alone, with the same saturated actor and no stability term. It shows how much variance the task reward leaves uncontrolled.
The practitioner baseline is manual reward engineering itself: SAC with a quadratic state penalty, or with tilt and torque penalties for locomotion. Its weights are tuned by grid search with a fixed budget of runs, equal to the total compute our method uses across its own seeds. Beating it means hitting the stated budget across a grid of noise levels and actuator limits without retuning, while the penalty baseline either needs retuning per setting or violates the budget.
The strongest learning competitor is a constrained-MDP version of SAC. It uses a Lagrangian multiplier on the episodic average of $\|x_t\|^2$, constrained to be at most $B$, with the same networks, data, seeds and multiplier learning rate. This is the most direct simpler route to "a variance number instead of a weight," which is why we treat it as the main threat.
The key ablation is LAC exactly as in Reinforcement Learning for Control with Probabilistic Stability Guarantee, which is our method with $b=0$ and otherwise identical. It isolates whether the offset matters under unbounded noise. No numerical cost or death-rate values for LAC were found on Lacuna; the file reports only qualitative superiority over SAC, SPPO and LQR.
On linear problems, stochastic MPC is the gold standard. Its reported result on its 3D example is a state norm "averaging 10-20 units" over 200-step realizations under high-variance Gaussian noise (Stochastic receding horizon control with output feedback and bounded control inputs). We count it as matched if the learned policy reaches comparable long-run variance with the same actuator bound and no model.
For safety filters, we wrap the unshaped policy in the variance-buffered barrier controller of Robust Safety under Stochastic Uncertainty with Discrete-Time Control Barrier Functions. That controller reduced violations from 6 (standard DTCBF) to 1 on a Unitree A1 narrow path. Domain randomization, which RRLS : Robust Reinforcement Learning Suite finds to be a strong average-case baseline, is applied on top of the penalty-SAC baseline with the same randomization ranges we use at test time.
Why not just…?
Why not just put a constraint on average squared state in a constrained MDP? This is the real threat. It also replaces weights with a budget, and on short evaluation horizons it may match us. The difference is what gets certified. An episodic average constrains the states the policy visits within the training horizon. It says nothing about the long-run moment beyond that horizon, or from initial states the policy rarely saw. The drift condition, by contrast, is a one-step inequality whose consequence extends to infinite time. We settle the question by evaluating at ten times the training horizon and from shifted initial-state distributions. If the constrained-MDP baseline keeps its budget there as well, our contribution shrinks to the feasibility diagnostic and the interpretable critic, and we will report that.
Why not just tune a quadratic penalty, since a Lagrange multiplier is just an automatically tuned weight? At a single operating point this is true by duality. The question is whether the right weight transfers. We tune the penalty at one noise level and one actuator limit, then test across a grid of both. If one weight keeps the variance under budget everywhere, the method has no advantage beyond convenience. The weeks of tuning reported for LittleDog in Hierarchical Apprenticeship Learning with Application to Quadruped Locomotion suggest it will not.
Why not just shield an unshaped policy with a stochastic barrier filter? The Jensen-enhanced barrier controller handles unbounded noise for set-exit probabilities over a finite horizon. However, it assumes a known noise covariance and a hand-designed barrier, and it does not shape the learned policy. As a result, the learner may rely on the filter constantly. We measure how often the filter intervenes and the resulting jitter. A shielded policy that matches our variance with few interventions would be a negative result for us.
Why not just use domain randomization? RRLS : Robust Reinforcement Learning Suite shows that it helps average-case returns. It does not specify any variance target, though, so it still needs the penalty weights it is meant to make robust. We include it as an add-on to the penalty baseline, not as a substitute for the budget.
Proposed Experiments
Linear systems. We use the 3D saturated linear system of Stochastic receding horizon control with output feedback and bounded control inputs and the 4th-order integrator of On stochastic MPC formulations with closed-loop guarantees: Analysis and a unifying framework. We add Gaussian noise and sweep $U_{\max}$, and we also include one open-loop unstable variant.
Nonlinear benchmarks. We use stabilization tasks for cart-pole and 2D quadrotor in safe-control-gym, with white-noise disturbance injection and actuator limits. The file gives no reported RL variance numbers there; the only reported figure is GP-MPC's roughly tenfold data advantage over PPO and SAC, which we note as context.
Locomotion. We use PyBullet Minitaur from the LAC suite with random pushes. No reported numerical result was found on Lacuna for Minitaur.
Metrics. Long-run variance is the Monte Carlo estimate of $\mathbb{E}\|x_t\|^2$ over the final half of rollouts ten times the training horizon, using 500 rollouts as in the barrier-function study. We also measure the fraction of noise/actuator settings in which the budget is met without retuning, the number of tuning runs a practitioner spends, task return, the fraction of time at actuator saturation, and action jitter.
Decisive checks. There are three. First, on linear problems, the method's certified minimal budget should track the stochastic-MPC variance and the analytic actuator thresholds. It should fail loudly, with a diverging multiplier, on the unstable plant instead of reporting false success. Second, the $b=0$ ablation should degrade sharply as noise grows while the offset version holds its budget. Third, against the constrained-MDP baseline at ten times the training horizon and under initial-state shift, our method must keep its budget where the baseline drifts. If it does not, the core claim fails.
References
- Stochastic receding horizon control with output feedback and bounded control inputs
- Mean-square boundedness of stochastic networked control systems with bounded control inputs
- Reinforcement Learning for Control with Probabilistic Stability Guarantee
- Robustifying Model Predictive Control of Uncertain Linear Systems with Chance Constraints
- Stochastic Model Predictive Control for Linear Systems using Probabilistic Reachable Sets
- Hierarchical Apprenticeship Learning with Application to Quadruped Locomotion
- Learning Agile Locomotion via Adversarial Training
- Robust Safety under Stochastic Uncertainty with Discrete-Time Control Barrier Functions
- Rrls: Robust reinforcement learning suite
- On Stochastic MPC Formulations with Closed-Loop Guarantees: Analysis and a Unifying Framework
- Safe-Control-Gym: A Unified Benchmark Suite for Safe Learning-Based Control and Reinforcement Learning in Robotics