Self-Tuning Constraint Penalties for Reward-Free Robot Training

Training a legged robot with reinforcement learning usually means writing one reward that adds forward speed to a dozen hand-weighted penalties for torque, foot slip, and body tilt. An engineer then re-tunes those weights every time the robot, the terrain, or the target speed changes. A cleaner option is to state each penalty as a budget in physical units, such as "average torque below 25% of maximum," and let a constrained optimizer find the trade-off. That shifts the burden instead of removing it. Constrained reinforcement learning methods have their own sensitive knobs: the multiplier learning rate, or three control gains. Badly set knobs cause the safety cost to swing around its budget instead of settling.

This note proposes to remove those knobs too. It borrows a trick that already works in classical optimization. The Alternating Direction Method of Multipliers (ADMM) splits a hard constrained problem into simpler alternating steps, and it is notoriously sensitive to its single "penalty" constant. Adaptive ADMM with Spectral Penalty Parameter Selection fixed this by measuring the local curvature of the problem from recent iterates and setting the penalty from it. A built-in reliability check skips the update whenever that measurement is untrustworthy. We carry this rule into constrained policy training. Each physical constraint gets its own penalty, set automatically from how strongly that constraint's cost has actually responded to past changes in its multiplier.

Core Claim

In constrained reinforcement learning, the penalty coefficient of an augmented Lagrangian can be estimated online, per constraint, from observed data. An augmented Lagrangian is the usual Lagrangian plus a quadratic term that punishes violations more the larger they are. We estimate its coefficient as a quasi-Newton step on the dual problem. The estimate comes from the slope between multiplier changes and constraint-cost changes, and it is applied only when that slope is statistically reliable.

Our central claim is about robustness. One configuration of the resulting method should hold every constraint near its budget, without oscillation, across tasks, reward scales, initial penalties spanning several orders of magnitude, and changes in robot dynamics. The strongest existing stabilizer, PID Lagrangian control, needs re-tuned gains to do the same.

The new piece is not the augmented Lagrangian itself. Dual Optimistic Ascent (PI Control) is the Augmented Lagrangian Method in Disguise already shows that a fixed-penalty augmented Lagrangian and proportional-integral multiplier control coincide in the single-step regime. Any fixed-penalty version of our idea is therefore roughly PID Lagrangian. What is new is the automatic, curvature-based, safeguarded choice of that penalty, made separately for each constraint and in the noisy, nonconvex setting of policy optimization. Classical adaptive ADMM has never been tested in that setting.

Research Program

We cast each task as a constrained Markov decision process: maximize task return subject to $m$ cost budgets $C_i(\pi) \le B_i$, where $\pi$ is the policy and $C_i$ is the expected cost of constraint $i$. Following ADMM's variable-splitting recipe, we introduce a nonnegative slack $s_i$ per constraint, giving the equality $C_i(\pi) - B_i + s_i = 0$.

One outer iteration has three steps. The policy step runs a few PPO epochs on the augmented Lagrangian, using a learned cost critic per constraint. The slack step is a closed-form projection, $s_i = \max(0, -(g_i + \lambda_i/\rho_i))$, where $g_i = \hat C_i - B_i$ is the measured violation. The dual step is $\lambda_i \leftarrow \lambda_i + \rho_i (g_i + s_i)$, where $\rho_i$ is constraint $i$'s penalty. The policy step uses several inner epochs, which is exactly the regime where the PI/augmented-Lagrangian equivalence of Dual Optimistic Ascent breaks down. Whether the quadratic term then helps beyond PID damping is an empirical question this program answers.

The adaptive part works per constraint, over a sliding window of the last $W$ outer iterations. In that window we record changes in multiplier $\Delta\lambda_i$ and in measured violation $\Delta g_i$. The dual function's gradient is the violation, so the negative slope of violation against multiplier is the dual curvature. A Barzilai-Borwein estimate of it gives a Newton-like step size:

$$\rho_i = \frac{\langle \Delta\lambda_i, \Delta\lambda_i\rangle}{-\langle \Delta\lambda_i, \Delta g_i\rangle}, \qquad \text{applied only if } \; r_i = \frac{-\langle \Delta\lambda_i, \Delta g_i\rangle}{\|\Delta\lambda_i\|\,\|\Delta g_i\|} > \epsilon^{cor}.$$

The window-averaged estimate in the first expression is the Barzilai-Borwein step that makes a multiplier change cancel the observed violation in one step. In the second expression, $r_i$ is the correlation between the two series and $\epsilon^{cor}$ is the threshold from Adaptive ADMM, about 0.2.

In that paper the correlation test guarded against nonlinearity. Here it also acts as a noise filter. When rollout noise swamps the signal, the correlation drops and the penalty stays where it is.

Two further safeguards come from Adaptive Consensus ADMM, whose per-node penalties we reinterpret as per-constraint penalties. First, each update may change $\rho_i$ only by a bounded factor. Second, that bound shrinks over training. In the convex case this kind of bound is what preserves the $O(1/k)$ guarantee. Here it at least prevents feedback between penalty and policy from compounding.

The method has no reward-scale knob, because the estimate is a ratio in the units of each constraint. A torque cost and a slip cost therefore get penalties matched to their own sensitivities without anyone choosing them.

We will also analyze, in a tabular constrained MDP where the dual function can be computed exactly, whether the spectral estimate tracks the true dual curvature. The test environments are the gridworld settings of RCPO and VMDP. This analysis tells us whether the correlation test rejects updates for the right reasons.

Baselines

The naive reference point is the manual practice the proposal targets. A single scalar reward subtracts weighted costs, and the weights are found by grid search. We give the grid search a tuning budget equal to the total number of runs every other method consumes, so its cost is visible.

The textbook constrained baseline is Reward Constrained Policy Optimization. It updates the multiplier by plain gradient ascent on a slow timescale, and on its torque-constrained MuJoCo tasks it "stayed at or near the 25% torque limit" while fixed penalties either broke the constraint or lost return. No exact return figure appears in the file.

The strongest competitor is PID Lagrangian. It adds proportional and derivative terms to the multiplier and rescales gradients so that hyperparameters survive reward scaling by 0.1 or 10. On Safety Gym, $K_P=1$ "significantly damps oscillations." No reported numeric return or cost for the Point, Car, or Doggo tasks was found on Lacuna, so we will reproduce it ourselves.

The PID gains will be tuned in two ways. In the first, they are tuned per task with the same run budget as the grid search. In the second, they are tuned once on Point-Goal and frozen. Our method gets no per-task tuning at all.

We also include CVPO, whose expectation-maximization formulation reportedly needs 100x to 1000x fewer samples than CPO. It is a different design point, off-policy and computationally heavy, so we compare it on both samples and wall-clock time.

Ablations isolate the key ingredient. Fixed-penalty augmented Lagrangian ADMM, with the penalty tuned, tests whether adaptation matters. Residual balancing in place of the spectral rule tests whether curvature is needed. Removing the correlation test tests the noise filter. A single shared penalty versus per-constraint penalties tests the decoupling.

Beating PID Lagrangian means one of two things. Either we match its tuned-per-task return and violation with zero tuning, or we strictly beat its frozen-gain version on violation overshoot and time-to-settle across the transfer conditions.

Why not just…?

Why not just tune the scalar reward weights with grid search or population-based training? Tuned weights are valid for one dynamics setting. RCPO's Ant results show fixed penalties either violating the torque budget or collapsing return. The cost-of-transport gap in Adversarial Motion Priors (0.93–1.12 versus 1.37–1.65 for hand-designed rewards) shows how much hand weighting leaves on the table. Our dynamics-shift experiment makes this concrete. We tune weights at nominal mass and friction, then measure violation after a change; we expect the weights to need re-tuning while budgets do not.

Why not just use PID Lagrangian, which already damps oscillations and is scale-invariant? This is the real threat. If frozen PID gains transfer across tasks, constraint counts, and dynamics as well as our method does, the remaining advantage shrinks to per-constraint decoupling. Its scale invariance acts on the reward-to-cost gradient ratio, not on differences between constraints, and its derivative term needs smoothing under rollout noise. The frozen-gain transfer comparison and the multi-constraint locomotion tasks are designed to settle this. We will report the result whichever way it falls.

Why not just adapt the ADMM penalty by residual balancing, the classical heuristic? Adaptive ADMM reports that residual balancing is not scale-invariant and degrades under unit changes. It also reports that vanilla ADMM failed on 13 of 23 problems where spectral adaptation converged on all. ADMM without a Fixed Penalty Parameter notes that residual balancing carries no rate-improvement guarantee. The residual-balancing ablation, run under cost-unit rescaling, tests whether this carries over to policy training.

Proposed Experiments

The experiments use three suites from the file. The first is Safety Gym (Point, Car, and Doggo robots on Goal and Button tasks), the shared ground of PID Lagrangian and CVPO. The second is the six torque-constrained MuJoCo tasks from RCPO, extended from one torque constraint to three: torque, joint-velocity, and body-tilt budgets. The third is the tabular gridworlds used for curvature validation. For multi-agent stress tests we will use Safety Multi-Agent MuJoCo through the MACPO codebase, whose MAPPO-Lagrangian serves as the multi-agent baseline.

Metrics are final return and cumulative training violation above budget. We also measure the peak overshoot and settling time of each cost relative to its budget, and the total number of runs spent on tuning.

The decisive checks are sensitivity sweeps. The first varies the initial penalty from $10^{-3}$ to $10^{3}$, mirroring AADMM's ten-orders-of-magnitude result. The second varies reward scale from 0.1 to 10. The third rescales each cost's units independently. The fourth varies robot mass and friction by ±30% and the target speed after any tuning is done.

The hypothesis is falsified if our curves vary across these sweeps as much as fixed-penalty ALM does. It is also falsified if frozen PID matches ours everywhere. Separately, the gridworld check fails if the spectral estimate does not correlate with the exact dual curvature.