Robots learn much faster when they choose among a few reusable motions ("reach here", "push there", "grasp this") instead of setting motor commands every few milliseconds. Each motion is an action primitive: a short, fixed controller that runs for hundreds of low-level steps once the learner picks its parameters. Safety, though, has to act at the millisecond level. A safety filter (often called a shield) watches every motor command and overrides it when the robot is about to hit something. The two clash. When the shield cuts a primitive off halfway, the planner that chose the primitive never learns why. It sees a lower task reward and no record of the near-miss, because the shield prevented the violation that would have produced a cost signal. The planner stays blind to where the danger lies, keeps proposing motions that get interrupted, and the robot moves in stop-start fits or freezes.
We propose to treat every shield intervention as a measurable cost that the primitive-choosing policy must keep within a budget. Each intervention also counts as a legitimate way for a primitive to end. The shield keeps the robot from violating constraints during training. The intervention budget teaches the planner to choose primitives and parameters that seldom need rescuing. The sparse task reward stays a simple success signal, and no one hand-tunes a penalty weight. If this works, primitive-based reinforcement learning (RL) can run with near-zero violations during training, without the timidity typical of flat constrained RL, and with fewer aborted motions than a shielded primitive learner that cannot see the shield.
Core Claim
The research file names an unresolved tension (its "Gap 4"). Primitives gain efficiency by committing to long macro-actions. Accelerating Robotic Reinforcement Learning via Parameterized Action Primitives (RAPS) runs each primitive for 100–300 low-level steps. Using these primitives instead of raw actions raised the number of Meta-World tasks solved from 21 to 43 out of 50. Per-step safety mechanisms work at the opposite timescale. One example is the neural Lyapunov shield in HiSaRL, which overrides the low-level policy when the robot leaves a learned stable region. Another is the projection layer in Constrained Markov Decision Processes via Backward Value Functions. Both must be able to step in at every timestep.
Our claim is that this clash is mainly a problem of information flow, not of architecture. A shield that hides its interventions from the high level recreates, one level up, the "safety paradox" described in Breaking Safety Paradox with Feasible Dual Policy Iteration: an agent kept safe gathers no evidence about where the boundary is. Exposing the interventions as a constrained cost at primitive-decision time closes the gap.
Formally, the high level picks primitive $k$ with parameters $\theta$ at decision time $t_j$. The primitive runs until it finishes or the shield aborts it. It returns a task reward $R_j$ (sparse success) and an intervention cost $C_j$, the number of low-level steps in which the shield changed or vetoed the command. The high-level problem is
$$\max_{\pi} \; \mathbb{E}\Big[\sum_j \gamma^{t_j} R_j\Big] \quad \text{s.t.} \quad \mathbb{E}\Big[\sum_j C_j\Big] \le d,$$
where $\pi$ is the primitive-and-parameter policy, $\gamma$ the discount factor and $d$ an intervention budget per episode. This is a constrained problem over decisions that each last a variable amount of time, a semi-Markov decision problem.
Following the backward-value idea of Constrained Markov Decision Processes via Backward Value Functions, we check a local condition at each decision point:
$$B(s_{t_j}) + Q_C(s_{t_j}, k, \theta) \le d,$$
Here $B$ is the expected intervention cost accumulated so far in the episode, and $Q_C$ is the predicted intervention cost of running $(k,\theta)$ now and following $\pi$ afterward. Candidate parameters that fail the condition are discarded before execution. This turns an episode-level budget into a decision-level filter applied only at primitive boundaries. The fast shield keeps sole authority over physical safety.
Research Program
The system has three parts.
The first is a parameterized primitive library in the RAPS style, kept fixed so that skill learning does not confound the comparisons. For navigation it holds move-to-waypoint, rotate and push. For manipulation it holds reach, grasp, lift and push.
The second is a low-level shield. The planner does not need to know how the shield works, only how often it acts. We use two shields to test that independence. One is a conservative braking filter built on the benchmark's cost function and a one-step predictive model. The other is a learned Lyapunov-style shield as in HiSaRL.
The third is the high-level learner. It is an option-level actor-critic with two critics, one for reward and one for intervention cost, plus a backward cost estimator over decision points. The dual variable on the budget $d$ is updated by primal–dual ascent, the same mechanism analyzed in Dong-Sheng Ding et al.. The decision-level filter above protects against the multiplier oscillation reported for Lagrangian methods in Safety Gymnasium.
When the shield aborts a primitive, the transition is stored with its true duration, the abort flag and the cost so far, and the high level chooses again from the resulting state. The planner therefore learns from what actually happened, not from what the primitive promised. This avoids the broken postcondition assumptions that the file flags for interrupted macro-actions.
On reward engineering, the proposal is deliberately minimal. Task reward is binary success, and the only safety quantity is the shield's own intervention count. Neither requires choosing a weight $\beta$ that trades progress against safety, which is the failure mode documented in Quanyi Li et al.. The budget $d$ still has to be chosen, but it has physical units ("rescues per episode") and we sweep it in the experiments. We do not claim to remove constraint engineering. The shield still needs a cost definition, and that is a real limitation the proposal shares with CNP3O.
The closest existing work combines some of these ingredients but never feeds the shield's behavior back to the primitive planner. Bridging the gap between Learning-to-plan, Motion Primitives and Safe Reinforcement Learning (CNP3O) optimizes B-spline primitive parameters with episodic PPO under adaptive Lagrange multipliers on analytical constraints. It has no runtime shield, so its constraints are satisfied only as the multipliers converge, not during exploration. Skill-based Safe Reinforcement Learning with Risk Planning (SSkP) plans over latent skills with a risk predictor learned from demonstrations and failure logs. It needs offline expert data and uses fixed skill horizons that cannot be interrupted. HiSaRL shields a fixed low-level controller under a geometric planner, and its planner does not learn from the shield's activity. Safe Policy Exploration Improvement via Subgoals (SPEIS) uses subgoals to cure the freezing of flat Lagrangian agents, but it acts on raw velocities and enforces safety only through the Lagrangian. Our contribution is the missing feedback channel and the budget accounting at decision points that it makes possible.
Baselines
The naive reference is a flat agent on raw actions: PPO, and PPO with a Lagrangian cost term (PPO-Lag), both from the SafePO library released with Safety Gymnasium. That paper reports that PPO-Lag cut safety violations by 98% relative to PPO on the velocity tasks, at the price of a 45% drop in reward. We run both with the library's default hyperparameters plus an equal-sized tuning sweep, on the same environments and the same number of environment steps (low-level steps, not decisions).
The strongest competitor on the question of interruption is shielded RAPS. It uses the same primitive library, the same shield and the same high-level learner, but the planner sees only task reward and the intervention signal is discarded. It shares every ingredient except the one we claim matters, so it doubles as the key ablation. Beating it means achieving at least equal success with a clearly lower abort rate and lower wall-clock time per episode, at matching (near-zero) true violation counts. Because these are new environment variants, no reported result exists on Lacuna for this baseline, and we will establish it ourselves.
A second strong competitor is a CNP3O-style constrained primitive learner. It uses the same primitives, a Lagrangian on true environment cost, and no shield. Here CNP3O reports a "near 100% scoring ratio" on its air-hockey task, but the file gives no figures on Safety-Gymnasium or Meta-World. The comparison point is cumulative violations during training. We expect it to match our final return but to incur many more violations along the way. For the narrow-corridor regime, SPEIS is the reference. SPEIS reports 84% success against 0% for SAC-Lagrangian on POLAMP Level 2, with roughly a 3% collision rate. We compare against those published numbers and against a rerun under our step budget.
Two further ablations isolate the remaining design choices. One removes the backward-value filter and keeps only the primal–dual multiplier, which tests whether filtering at decision points actually reduces oscillation. The other feeds interventions to the planner as a fixed reward penalty instead of a budgeted constraint (discussed below). Every method gets the same seeds (at least ten), the same total number of low-level steps and the same number of hyperparameter configurations.
Why not just…?
Why not just let the shield act silently and trust the high-level reward? In principle a shielded primitive is just a different primitive, and the planner's reward critic will eventually learn that interrupted motions pay less. This is the most serious threat to the proposal, and the shielded-RAPS ablation tests it directly. Our reason to expect a gap is that the shield's effect on sparse success reward is delayed and noisy, while the intervention count is immediate and dense. The planner should learn faster with the count, especially early in training. If shielded RAPS matches our abort rate and sample efficiency, the core claim is false and we will report it that way.
Why not just add a fixed penalty per intervention to the reward? This is the manual-penalty approach that Quanyi Li et al. show to be brittle: small weights are ignored and large ones produce "safe but useless" freezing. We sweep the penalty weight over a decade-wide grid. If some weight matches the constrained version across all tasks with a single setting, the budget machinery is unnecessary. We expect the best weight to differ between tasks, while a single budget expressed in rescues per episode should transfer.
Why not just use shorter primitives, or flat safe RL? Shortening primitives reduces the cost of each abort but gives up the exploration advantage that RAPS demonstrated: 43 versus 21 of 50 Meta-World tasks. Flat Lagrangian agents show the freezing reported by SPEIS (0% on POLAMP Level 2) and the 45% reward loss reported in Safety Gymnasium. We include a primitive-horizon sweep (25, 50, 100 and 200 steps) so that the shortening alternative is tested empirically, not dismissed.
Proposed Experiments
We use three testbeds. The first is Safety-Gymnasium navigation (Goal, Button and Push at level 2 with the Point and Car robots), with the primitive library wrapped around the native controllers. The second is the POLAMP narrow-corridor Level 2 task, which probes timidity. The third is a Meta-World subset with RAPS primitives and added forbidden regions near the workspace (for example a "do not touch" box beside a drawer), where the true cost is contact with that region.
Metrics are task success or return, cumulative true constraint violations summed over all of training, the shield intervention rate, the fraction of primitives aborted, and wall-clock time per episode. The last captures the stop-start motion problem.
Three outcomes would settle the question. First, against shielded RAPS at equal violations, our method should show a lower abort rate and higher success early in training, with the gap growing as hazards become denser. Second, against the CNP3O-style learner, our method should reach similar final success with substantially fewer violations accumulated during training. Third, against the penalty sweep, one budget value should work across tasks where no single penalty weight does. Swapping the braking shield for the learned Lyapunov shield should leave the ranking unchanged. If it does not, the method depends on the particular shield, and we will report it as a limitation.
References
- Accelerating Robotic Reinforcement Learning via Parameterized Action Primitives
- HiSaRL: A Hierarchical Framework for Safe Reinforcement Learning
- Constrained Markov Decision Processes via Backward Value Functions
- Breaking Safety Paradox with Feasible Dual Policy Iteration
- Natural Policy Gradient Primal-Dual Method for Constrained Markov Decision Processes
- Safety Gymnasium: A Unified Safe Reinforcement Learning Benchmark
- MetaDrive: Composing Diverse Driving Scenarios for Generalizable Reinforcement Learning
- Bridging the gap between Learning-to-plan, Motion Primitives and Safe Reinforcement Learning
- Skill-based Safe Reinforcement Learning with Risk Planning
- Safe Policy Exploration Improvement via Subgoals