Safe reinforcement learning (RL) usually asks an agent to maximize reward while keeping a long-run safety cost, such as collisions or hazard entries, below a budget. This setup is called a constrained Markov decision process (CMDP). The standard way to solve it adds the cost to the reward with an adaptive penalty weight, a Lagrange multiplier. Practitioners agree that this is fragile. If the penalty is set slightly too low, the agent is reckless. If it is set slightly too high, the agent freezes. Our hypothesis is that much of this fragility comes from the opening phase of training. In that phase the agent explores a huge action space, its cost estimates are noisy, and the multiplier chases that noise.
We propose to start the agent on a small, coarse set of actions, for example three steering values instead of seventeen. The agent stays there until it reliably meets the safety budget. Only then does the action set get finer, with each new action nested inside an old one, so that every refinement step leaves the agent's behavior exactly unchanged. The claim we will test is not mainly that this trains faster. The claim is that it shrinks the hyperparameter sensitivity of constrained RL: a single fixed configuration should work across environments, measured with the formal sensitivity metrics of A Method for Evaluating Hyperparameter Sensitivity in Reinforcement Learning.
Core Claim
The fragility of Lagrangian methods is documented but not measured. Safety Gymnasium benchmarks 16 safe RL algorithms on 54 tasks and reports the "too scared to move" versus reckless failure modes, but only qualitatively. CHPO drops Lagrangians in hybrid discrete-continuous action spaces for exactly this reason. It replaces them with a switch: when cost is over budget, the update minimizes cost; otherwise it maximizes reward. That switch introduces its own brittle knob, the fraction of updates spent on cost.
Adkins et al. give a way to price this fragility with two quantities. Sensitivity $\Phi$ is the performance lost by using one best fixed configuration everywhere instead of tuning per environment. Effective dimensionality $d$ is how many hyperparameters must be retuned per environment to reach 95% of fully tuned performance. They measured these only on unconstrained continuous control.
We claim that a safety-gated, lossless, coarse-to-fine action curriculum will lower both $\Phi$ and $d$ for Lagrangian CMDP solvers without lowering final constrained performance. We will also identify which part of the curriculum is responsible: the inherited policy, the inherited multiplier, or the reduced early exploration.
How the curriculum works
We define a nested sequence of discrete action sets $A_1 \subset A_2 \subset \dots \subset A_J$. For the continuous parts of an action, each dimension is binned with odd counts such as 3, 5, 9 and 17, so each refinement adds midpoints and keeps the old bins. For discrete parts, such as CHPO's actuator switches or parking curves, the agent starts with a subset of the options. Every fine action $a'$ has a parent $p(a')$ in the previous set, chosen as the nearest coarse action.
When the stage advances, the new policy is initialized as follows, where $C(p(a'))$ is the set of fine actions sharing that parent:
$$\pi_{j+1}(a' \mid s) = \frac{\pi_j(p(a') \mid s)}{|C(p(a'))|}.$$
This spreads each coarse action's probability evenly over its children. The executed behavior is essentially preserved, and the Lagrange multiplier carries over unchanged.
Advancement is gated by safety, not by reward progress. The agent moves to the next stage after its undiscounted episodic cost has stayed under budget for $K$ consecutive evaluations. CHPO showed that undiscounted cost is needed to catch delayed hazards, so we use it here too.
This design differs from its closest relative, Mix&Match. Mix&Match bootstraps a 756-action agent from a 9-action agent in unconstrained DeepMind Lab, using policy mixing, distillation, and Population-Based Training to schedule the mixing weight. Our nesting makes the transition lossless, so distillation and a scheduling population are unnecessary. The gate is defined by the constraint, and the object of study is tuning sensitivity, which Mix&Match never measured.
Relation to safe curricula
Two existing curricula target training-time safety by changing the outside world.
Safe Reinforcement Learning via Curriculum Induction (CISR) trains a teacher. The teacher picks reset interventions over generations of student agents and cuts Lunar Lander training failures from over 1,200 to fewer than 3. Safety-Prioritizing Curricula for Constrained Reinforcement Learning (SCG) shifts the task distribution from safe contexts toward the target task and reaches 100% target success with the lowest violation regret on its navigation suite.
Both papers motivate our work, because they show that curricula reduce violations. Both also constrain it. SCG warns that reward-driven curricula push agents through hazards, which is why our gate uses cost. Both curricula need environment-side control: a reset library or a configurable task distribution. Ours needs only control over the agent's own action interface, so it applies when the environment is fixed. Neither paper asks whether its curriculum changes how much tuning the underlying solver needs.
Research Program
The study has three parts.
First, we produce the missing measurement: $\Phi$ and $d$ for constrained solvers. Constrained performance needs a scalar score. Our primary score is normalized reward on evaluation episodes that meet the cost budget, with zero for violating runs. We normalize per environment with Adkins' [5, 95] percentile scaling. We also report raw reward, final cost, and cumulative training cost, so that no conclusion depends on the choice of scalarization.
Second, we compute the same metrics with the action curriculum wrapped around each solver.
Third, we run ablations that separate the three candidate mechanisms behind any gain:
- Inherited multiplier only: the agent trains from scratch on the fine action set, starting from the multiplier value the curriculum reached.
- Inherited policy, reset multiplier: the agent keeps the expanded policy but restarts the multiplier.
- Ungated curriculum: stages advance on a fixed time schedule rather than the safety gate.
If the multiplier-only ablation recovers the whole benefit, the honest conclusion is that constrained RL's tuning tax is a multiplier-initialization problem. That finding is useful in its own right, and we will report it as such.
Compute is the obvious obstacle, since Adkins et al. used 4.3 million runs. We shrink the problem deliberately. The grid covers four hyperparameters known to matter for Lagrangian methods: policy learning rate, multiplier learning rate, initial multiplier, and entropy coefficient. Each takes three or four values, about 144 configurations. We run 5 seeds on roughly eight small environments. This gives a few thousand short runs per method, which is feasible on cluster-scale CPU simulation. Adkins et al. note that $\Phi$ and $d$ depend on grid bounds and on the suite, so every method uses identical grids, and we report the metrics' stability under grid subsampling.
Baselines
Naive reference point. Plain PPO-Lagrangian is trained directly on the finest action set with no curriculum, using the SafePO implementation from Safety Gymnasium. On velocity tasks it is reported to cut violations by 98% relative to PPO at a 45% reward cost. It defines the tuning tax we aim to remove. Beating it means a lower $\Phi$ and a lower $d$ at equal or better constrained score.
Stabilized-multiplier baseline. CPPO-PID, from the same suite, replaces the multiplier update with a proportional-integral-derivative (PID) controller. The suite describes it as one of the most effective stabilizers. It is the strongest direct competitor on our specific question, since it also targets multiplier oscillation. Its three controller gains join the tuning grid, which is the fair cost of using it. We beat it if the curriculum applied to plain PPO-Lagrangian reaches lower $\Phi$ than CPPO-PID. The comparison counts the PID gains as tunable hyperparameters in $d$.
Projection-based reference. PCPO projects each update onto the linearized constraint. Safety Gymnasium reports it as stable but often overly cautious. It is the low-sensitivity, low-reward corner that we must not simply reproduce.
Strongest hybrid-action baseline. On CHPO's hybrid benchmark (Moving, Sliding, HardMove, Parking, cost budget 1.0), the strongest method is CHPO itself. It reports a Parking reward of 37.63 against 20.01 for HPPO-Lag, and it is the only method that stayed within budget on all four tasks. We run CHPO with its cost-update fraction added to the grid. We beat it if our curriculum plus HPPO-Lag matches 37.63 within budget under a single fixed configuration across the four tasks, or if CHPO's own $\Phi$ drops when the curriculum is wrapped around it.
Environment-side safe curriculum. SCG runs on its safety-maze and safety-goal tasks. Its paper reports 100% target success and the lowest regret, but the file gives no numeric regret value: no reported result found on Lacuna for SCG's exact cumulative violation regret. We will therefore re-measure SCG's regret under our grid.
Matched-budget rule. All methods receive the same configuration grid, seeds, and environment-step budget. Curriculum stages count against that budget.
Why not just…?
Why not just train on the coarse action set forever? If three steering values are safe and good enough, no curriculum is needed. Two pieces of evidence argue against this. The discretization literature summarized in Overcoming Discrete Action Space Constraints in Robotics shows that coarse sets cap tracking and efficiency. Mix&Match found that the large action space ultimately reached higher return. We include coarse-only training as a baseline. If it matches the final constrained score on most tasks, the curriculum is unjustified there, and we will say so.
Why not just anneal the cost budget instead, starting loose and tightening? This is the most serious threat. It is a one-knob curriculum that needs no action engineering. Its weakness is that a loose budget teaches the multiplier the wrong scale, and it still accumulates violations early, which is the failure SCG describes for safety-blind schedules. We run budget annealing under the same grid. If it achieves equal $\Phi$, the paper's conclusion becomes that any curriculum which delays tight constraint pressure removes the tuning tax. That would be a weaker claim, but a true one.
Why not just use a PID Lagrangian? CPPO-PID damps oscillation but adds three gains. Adkins et al. showed that "stabilizing" tricks like advantage normalization can raise sensitivity even while raising peak performance. Whether PID lowers or raises $d$ is unmeasured. The CPPO-PID baseline answers this directly, and wrapping the curriculum around CPPO-PID tests whether the two effects combine.
Why not just warm-start the multiplier? If the benefit comes only from a sensible initial multiplier, a cheap pilot run could supply one. The multiplier-only ablation tests exactly this. We do not presume the answer.
Why not just use Mix&Match with a Lagrangian? Mix&Match is the existing action-space curriculum, but it relies on a Population-Based Training population to schedule its mixing weight. That is itself tuning compute and so hides the tax we want to measure. We run a Lagrangian Mix&Match variant under the matched budget, counting its population members as configurations.
Proposed Experiments
The environments are three Safety Gymnasium navigation tasks (Point and Car goal tasks) with factorized discretization at resolutions 3, 5, 9 and 17 per dimension; CHPO's four hybrid tasks, coarsened over curves, switches and distances; and SCG's safety-maze and safety-goal.
The first decisive check is whether $\Phi$ for curriculum-wrapped PPO-Lagrangian falls below that of both plain PPO-Lagrangian and CPPO-PID, with confidence intervals from bootstrapping over seeds and grid subsamples, while the best-tuned constrained score stays within noise.
The second check is whether $d$ drops, for example from needing per-environment multiplier and learning-rate tuning to needing only one or neither.
The third check is cumulative training cost relative to SCG and to budget annealing.
The ablations then attribute whatever gain appears. We will place each method on Adkins' performance-sensitivity plane. That plane would be the first such map for constrained RL, and it is publishable even if the curriculum loses.
References
- A Method for Evaluating Hyperparameter Sensitivity in Reinforcement Learning
- Safety Gymnasium: A Unified Safe Reinforcement Learning Benchmark
- CHPO: Constrained Hybrid-action Policy Optimization for Reinforcement Learning
- Mix&Match - Agent Curricula for Reinforcement Learning
- Safe Reinforcement Learning via Curriculum Induction
- Safety-Prioritizing Curricula for Constrained Reinforcement Learning