Engineers who train robot hands and humanoids with reinforcement learning spend much of their time on reward weights. They add a penalty for torque, a penalty for contact force, and a bonus for moving like motion-capture data, then retune the multipliers until the behavior looks right. This proposal replaces those weights with budgets stated in physical units, such as "average contact force at most $d$" or "style score at least $d$." It trains a single policy that can meet any such budget exactly, without retraining, by randomly switching between two of its own trade-off settings during an episode.
The theory of constrained decision problems explains why randomization is needed. With one constraint, the best policy is in general a random mix of two simpler policies. A fixed-weight reward can only produce the extreme points of what is achievable, so it overshoots or undershoots a budget that falls between them. That theory has only been put into practice with small discrete action sets, a tabular advertising task, and two-dimensional navigation. It has never reached the 24- to 56-joint systems where reward engineering hurts most. We also ask a question the literature has skipped: at what timescale should the random mixing happen, and does the answer matter for a physical robot?
Core Claim
A single actor conditioned on the penalty multiplier $\lambda$ can be trained once with $\lambda$ resampled at random within each episode. At deployment, its expected cost can then be driven to any target budget $d$ by switching between two multiplier values $\lambda_{lo}$ and $\lambda_{hi}$ with probability $t$ at each action-chunk boundary, where $t$ is found by bisection. Because the switching happens many times per episode rather than once per episode, the cost of each individual episode concentrates near $d$. Episode-level mixing, which is the construction existing methods use, instead gives a two-peaked cost distribution in which a fraction $t$ of episodes run over budget. We claim this recipe removes the weight-tuning loop in high-degree-of-freedom control and matches or beats Lagrangian methods at equal compute. We also claim the within-episode switching is what makes budgets usable on hardware, not the randomization per se.
The deployed objective is simple. Write $J_r(t)$ and $J_c(t)$ for the expected task return and expected constraint cost when the policy $\pi(a \mid s, \lambda)$ switches between $\lambda_{lo}$ and $\lambda_{hi}$ with probability $t$ per chunk. We then solve
$$\max_{t \in [0,1]} J_r(t) \quad \text{subject to} \quad J_c(t) \le d,$$
which is a one-dimensional search once the conditioned policy exists.
The claim does not presuppose that randomization helps in high dimensions. A Gaussian actor is already stochastic, and its achievable return-cost set may be close to convex. Measuring the size of this "hull gap" is part of the contribution. If the gap is negligible, that is a useful negative result for the safe-RL community.
Relation to Prior Work
Several papers motivate the need for randomization but stop short of high-dimensional control.
Budgeted Reinforcement Learning in Continuous State Space appends the budget to the state and mixes actions over a convex hull. This lets one policy span the Pareto front, the curve of best achievable return at each cost level. However, it requires small discrete action sets, and its budgeted Bellman operator is not a contraction. We condition on the multiplier $\lambda$ rather than the remaining budget, which avoids that operator entirely.
Constrained Reinforcement Learning via Policy Splitting finds an over-budget and an under-budget policy by sweeping fixed multipliers, then bisects a mixing weight. It was tested only on a 14-state, two-action advertising problem. We reuse its bisection step but obtain both endpoint policies from one conditioned network in continuous high-dimensional control.
Flipping-based Policy for Chance-Constrained Markov Decision Processes proves that flipping between two actions suffices for chance-constrained problems. Its practical version trains several networks at different budgets, uses a constant global flip weight, and was evaluated on Point and Car navigation. We make the flip cheap, since it is only a change of conditioning input, and we study its timescale.
Two lines of work supply the setting and constrain the design. ReLOAD shows that primal-dual training oscillates in its last iterate on continuous control up to a 56-DoF humanoid, and fixes this with optimistic updates on a unimodal Gaussian actor. Its thresholds and costs remain hand-set, and it does not offer a runtime budget knob. Safety Gymnasium provides dual Shadow Hand safety tasks and reports that Lagrangian methods are sensitive to multiplier tuning. Both papers are the targets our comparisons must beat.
On the reward-engineering side, Adversarial Motion Priors Make Good Substitutes for Complex Reward Functions replaces dozens of hand-tuned locomotion penalties with a learned style reward from a discriminator, a network trained to tell robot motion from motion capture. It still mixes style and task reward with fixed weights. We turn the style reward into a budget.
ADEPT and Towards Human-Level Bimanual Dexterous Manipulation with Reinforcement Learning establish that high-DoF hands are trainable with careful recipes. ADEPT delegates safety to an external reactive controller rather than a learned budget, which leaves open the question we address.
Research Program
The core artifact is a multiplier-conditioned PPO actor-critic built on the OmniSafe codebase (OmniSafe). Here $\lambda$ is fed to both networks. The critic has separate heads for reward and cost, so that $J_r$ and $J_c$ can be estimated for any $\lambda$ without retraining.
During training, $\lambda$ follows a random switching process. At each action-chunk boundary, with some probability, a new value is drawn from a log-uniform range. The value estimates therefore account for future switches instead of assuming $\lambda$ stays fixed forever. This is the step that makes within-episode switching legitimate at deployment. Without it, a policy trained under a constant $\lambda$ can behave badly when its conditioning changes mid-motion, for example by releasing a grasp. We test this directly with an ablation.
At deployment, we first evaluate the conditioned policy on a grid of $\lambda$ values to trace the lower convex hull of return versus cost. For a target $d$, we pick the two adjacent hull vertices and bisect $t$ using rollout estimates of $J_c(t)$. Since $J_c(t)$ is not guaranteed to be monotone under switching, we also measure how often bisection fails.
Three cost types cover the reward-engineering problem:
- Physical safety. Safety Gymnasium's collision and velocity costs.
- Effort. Summed joint torque, or contact-force excess over a threshold.
- Style. The negative of an adversarial-motion-prior discriminator score. It is constrained rather than weighted, which directly replaces the $w_s r_s + w_t r_t$ sum.
The analysis has two parts. First, we quantify the hull gap: the difference between the best return reachable by any single fixed $\lambda$ that satisfies $d$ and the return of the mixed policy, reported as a function of $d$ and action dimension. Second, we characterize the per-episode cost distribution as a function of switching granularity, from once per episode, to per chunk, to per step. We want to see where jitter at contact transitions starts to hurt task success, which is the practical limit on fine-grained switching.
Baselines
Unconstrained PPO with the benchmark's default hand-shaped reward. This is the do-nothing reference. It sets the maximum-return, uncontrolled-cost corner of the plot, and it runs with the same environment steps and seeds as every other method.
Hand-tuned scalarization. This is the practice we aim to replace: a separate PPO run for each fixed penalty weight on a grid, with the designer choosing the run that satisfies the budget. We give this baseline the same total environment-step budget as our single conditioned run, divided across its grid. We also report the number of runs it needed. We beat it if we match its return at the budget with a single run, or exceed it at equal steps.
Adaptive-multiplier methods. PPO-Lagrangian and the PID-controlled Lagrangian (CPPO-PID) from SafePO. On velocity tasks, PPO-Lagrangian is reported to give a 98% reduction in safety violations with a 45% drop in total reward relative to PPO (Safety Gymnasium). No dexterous-hand numbers are reported there; no reported result found on Lacuna for Safety-DexterousHands returns or costs.
ReLOAD. The strongest existing method, because it targets exactly the last-iterate instability that makes Lagrangian methods unreliable at a fixed budget in high-DoF control. We reimplement its optimistic update inside the same PPO code with the same network sizes and an equal hyperparameter search budget. Its paper reports only qualitatively that it was the only method to achieve high reward while obeying the humanoid height limit. No numerical return or cost is reported, so no reported result found on Lacuna for ReLOAD's Humanoid numbers.
Because ReLOAD and the Lagrangian methods require a fresh run for each budget, we compare at three budgets per task. Our single run is charged its full cost at each of those three budgets. Beating ReLOAD means equal or higher return at a matched expected cost, with smaller per-episode cost spread, while using one training run instead of three.
Flipping-based policy. On PointGoal2 and CarGoal2 we run the flipping-based policy as a sanity check that our method reproduces the known advantage of randomization in low dimensions. The only numerical result the file gives for it is from a point-mass toy task: mean reward 1.8259 versus 0.8667 for a deterministic policy at an identical 17% violation probability (Flipping-based Policy). For the Safety Gym tasks, no reported result found on Lacuna beyond "higher rewards at the same cost limit" against CPO, PCPO, P3O and CUP.
Ablations. Two ablations isolate the key ingredient. The first uses our conditioned policy with episode-level mixing, which is the construction of Chen et al. and Shen et al. The second applies within-episode switching to a policy trained with constant $\lambda$, isolating the switching-aware training.
Why not just…?
Why not just sweep reward weights and pick the one that fits? On a smooth return-cost curve this can come close, and our experiments report how close. The limits are that a deterministic scalarized optimum sits at a hull vertex, so intermediate budgets are either violated or undershot (ReLOAD), and that every new budget costs new runs (Budgeted RL). The equal-compute comparison settles whether this matters in practice. If the hull gap is small and a five-point sweep is cheap, we will say so.
Why not just use ReLOAD or a PID Lagrangian at the desired budget? These are real threats for a single fixed budget, and ReLOAD may well match our return there. Our advantage must come from amortization, meaning one run serving all budgets, and from tighter per-episode cost. The three-budget, equal-compute comparison and the per-episode cost-spread metric decide this.
Why not just flip a coin at the start of each episode between two separately trained policies? This achieves the expected budget exactly, and it is the strongest simple alternative. Its weakness is that a fraction $t$ of episodes run the over-budget policy start to finish. On a robot, that means some episodes exceed the force or safety limit every time. The episode-level-mixing ablation measures exactly this. If per-episode spread is no worse than under chunk-level switching, our key ingredient fails and the proposal reduces to cheaper endpoint training.
Why not rely on the Gaussian actor's own randomness? The noise in a Gaussian actor is local jitter around one mean action, not a choice between two distinct behaviors. For example, it cannot represent "grasp firmly" versus "yield compliantly" without averaging them into an action that drops the object. How much this costs in 24-DoF hands is unknown. The hull-gap measurement answers it, and a near-zero gap would be reported as a finding.
Proposed Experiments
We start with a low-dimensional check on PointGoal2 and CarGoal2 to confirm that the conditioned-and-switched policy recovers the flipping-based advantage. We then run the main experiments on the Safety-DexterousHands tasks of Safety Gymnasium: dual 24-DoF Shadow Hands catching and passing objects, with collision and contact-force costs. These are extended with a torque-effort budget on two Bi-DexHands tasks. Next comes the Humanoid height constraint from the ReLOAD suite. Finally, AMP-style quadruped locomotion tests the style budget: we constrain the discriminator score rather than weighting it and compare against the fixed-weight AMP configuration. That configuration is reported to give a Cost of Transport of 0.93 to 1.12 versus 1.37 to 1.65 for hand-designed rewards (AMP), and we check whether a style budget reproduces this range without weight search.
For every method we report:
- task return at matched expected cost,
- absolute budget error $|J_c - d|$,
- the fraction of episodes exceeding $1.2d$,
- task success rate,
- training runs and environment steps consumed.
Each result uses five seeds.
Three results would be decisive:
- Hull gap. If the gap is below seed noise on the dexterous tasks, randomization is unnecessary in this regime and the paper becomes a negative result about constrained high-DoF control.
- Per-episode cost concentration. If chunk-level switching does not concentrate per-episode cost relative to episode-level mixing, the central mechanism fails.
- Switching-aware training. If removing switching-aware training does not raise dropped-object or fall rates under within-episode switching, that training component is unneeded and should be cut.
References
- Budgeted Reinforcement Learning in Continuous State Space
- Constrained Reinforcement Learning via Policy Splitting
- Flipping-based Policy for Chance-Constrained Markov Decision Processes
- ReLOAD: Reinforcement Learning with Optimistic Ascent-Descent for Last-Iterate Convergence in Constrained MDPs
- Safety Gymnasium: A Unified Safe Reinforcement Learning Benchmark
- Adversarial Motion Priors Make Good Substitutes for Complex Reward Functions
- ADEPT: Accelerating Dexterity via Pre-Training and Post-Training using Reinforcement Learning
- Towards Human-Level Bimanual Dexterous Manipulation with Reinforcement Learning
- OmniSafe: An Infrastructure for Accelerating Safe Reinforcement Learning Research