Robots that learn by trial and error need two things at once. They must plan over long horizons, for example finding a route through a maze with a bottleneck. They must also avoid damaging themselves while they learn. Hierarchical reinforcement learning handles the first need by splitting control into a "manager" that picks intermediate targets (subgoals) every few dozen steps and a "worker" that drives the motors to reach them. Safe reinforcement learning handles the second need by giving the agent a cost budget, such as a cap on how many hazard contacts are allowed per episode. These two lines of work rarely meet. When they do, the budget is enforced either by a hand-built safety shield or as a single total over the whole trajectory, and the subgoals are written in world coordinates, so every new room layout has to be relearned.
We propose that the manager hand the worker a slice of the safety budget along with each subgoal, and that the subgoal be expressed relative to the agent rather than in world coordinates. The worker then only has to answer a local question: "can I reach this displacement, given what I currently see around me, while spending at most $b$ units of cost?" That question depends on the nearby geometry, not on where the agent is in the world, so its answer should transfer across layouts. The manager keeps the global view of the map and the remaining budget, and its job becomes deciding how to spend the budget across a sequence of local hops. This split is the core of the proposal. The worker gets the spatial invariance of relative representations. The manager keeps the global context that over-localized representations are known to lose. The budget becomes a quantity passed down the hierarchy that can be checked at every hop.
Core Claim
The claim is that a budget-conditioned worker, trained on agent-centric observations and relative subgoals, learns a cost-to-reach function that stays accurate on unseen layouts. A manager that plans with that function will satisfy a trajectory-level cost budget during training, with fewer in-training violations than a flat or hierarchical Lagrangian agent. It will also transfer zero-shot to rearranged hazard maps where world-frame hierarchical agents fail.
Concretely, the worker learns a critic $C_\theta(o_t, \Delta g, b)$. Here $o_t$ is the agent-centric observation (lidar-style hazard distances and the agent's own velocities, with no absolute position), $\Delta g$ is the subgoal displacement written in the agent's frame, and $b$ is the budget slice. The critic predicts the cost the worker will incur while attempting the hop. A companion critic $R_\theta(o_t, \Delta g, b)$ predicts the probability of reaching the subgoal. The manager picks a sequence of hops $(\Delta g_k, b_k)$ subject to
$$\sum_{k} b_k \le z_t, \qquad C_\theta(o_{t_k}, \Delta g_k, b_k) \le b_k,$$
where $z_t$ is the remaining episode budget, depleted as costs occur (the Sauté-style budget variable), and $t_k$ is the time hop $k$ begins. Execution is "reach it within budget or abort": if the worker's realized cost on a hop exceeds $b_k$, the hop ends immediately and the manager replans with the smaller remaining $z_t$.
Where This Sits
Three hierarchical safe-RL systems supply the motivation. HiSaRL plans world-frame subgoals with A*/RRT* and guards the low level with a learned Lyapunov shield, reporting violations in under 0.3% of 10,000 test cases. However, its subgoals are tied to a global map and its manager is not learned. ACS projects each primitive action into a chance-constrained feasible set. It reaches near-zero in-training violations, but it has no temporally extended subgoals. STESS confines exploration to the null space of a prioritized inverse-kinematics hierarchy, which requires an analytical model and hand-placed envelopes. None of the three passes a budget from manager to worker, and none uses relative subgoals.
Two flat safe-RL methods supply our machinery. Sauté RL adds the remaining budget to the state, so that any off-the-shelf learner can satisfy constraints almost surely. Backward value functions turn a trajectory budget into a state-wise check of the form "cost so far plus expected future cost ≤ budget." We reuse both ideas, but at the level of the manager's hops, which neither paper considers.
On the hierarchical side, we borrow two mechanisms that were built for reachability, not safety. Strict Subgoal Execution terminates on subgoal failure to map where the worker can reliably reach. We extend that termination rule to budget overrun. HESS steers the manager toward novel yet reachable states. We keep a novelty bonus, but it is filtered by predicted cost. This matters because an unconstrained novelty drive pushes the agent into walls and hazards.
The case for relative representations comes from several flat or unconstrained settings. Observation Space Matters found that absolute position acts as a "malicious variable" that breaks generalization. Bimanual suture regrasping obtained robot-agnostic policies from relative poses. The UAS avoidance work used a goal-aligned agent-centric frame with a heuristic safety penalty in a flat policy. IARO discovered relative options, but with no safety component.
One known limitation constrains the design: over-localized states blind the policy to global structure. That limitation is why we keep the manager world-frame and make only the worker relative.
Research Program
The build has three parts. The first is the worker: a goal-conditioned soft actor-critic whose input is the agent-centric observation, the relative subgoal, and the budget slice. It carries two critics, cost-to-reach and success. Its training data is relabeled hops drawn from the agent's own experience. Budget slices are sampled from a range so that the worker learns the full trade-off between speed and cost rather than a single operating point.
The second is the manager. It sees the global position, the map, and $z_t$. It proposes candidate relative hops, scores them by estimated progress minus a novelty bonus, and prunes any candidate whose predicted cost exceeds the slice it would be given. Among the survivors, it allocates slices by a small search over a subgoal graph, in the spirit of SSE's frontier graph. The graph edge weights are predicted costs rather than predicted step counts.
The third is calibration of $C_\theta$. The allocation constraint is only as good as the cost critic. We will therefore study conservative estimates, specifically an ensemble upper quantile. We will also measure how often realized hop cost exceeds predicted cost, as a function of how far a test layout is from the training layouts.
The analysis centers on one question: does the cost-to-reach critic actually transfer? We will measure its error on held-out layouts under three input conditions: agent-centric input, world-frame input, and agent-centric input plus absolute position. The third condition deliberately re-injects the "malicious variable."
Baselines
The naive reference is a flat, unconstrained soft actor-critic on world-frame observations. It shows how much cost an agent accrues when nothing restrains it and how poorly it explores the long-horizon tasks.
The first strong flat competitor is PPO-Lagrangian from the SafePO library in Safety Gymnasium. That paper reports a 98% reduction in violations versus PPO, with a 45% drop in reward, on velocity tasks. We also run FOCOPS, which reports 1106.1 reward on HumanoidCircle at cost ≤ 50, compared with 671.0 for CPO. Both flat competitors are run on the same agent-centric observations we give our worker, so that representation is not confounded with hierarchy.
The second is flat Sauté-SAC with agent-centric observations. It is the most direct competitor on the safety question. For Sauté on maze-style tasks, no reported result was found on Lacuna.
The third, and the strongest on long-horizon success, is SSE. The paper reports near 100% success on AntDoubleKeyChest, where HIRO and HIGL score 0%, and 60% success on AntMazeBottleneck within 432 episodes. These figures are for the unconstrained setting. No reported result was found on Lacuna for SSE, or any hierarchical method, on AntMaze with hazard costs. We will therefore run SSE ourselves in two versions: as published, and with a Lagrangian penalty on total episode cost ("SSE-Lag").
The fourth is a classical pipeline in the style of HiSaRL: an A* planner on the global map feeding world-frame waypoints to a Sauté worker. It uses the same simulator, a known map, and no learned manager.
Two ablations isolate the key ingredients. The first gives the worker the whole remaining budget $z_t$ instead of an allocated slice, which removes budget allocation. The second gives the worker world-frame subgoals and observations, which removes relative representation.
For fairness, all methods get the same environment steps and the same network sizes. Hyperparameters are tuned with an equal budget of 20 configurations per method on the training layouts only. Every run uses five seeds.
Beating the field means two things. First, cumulative in-training cost and final success must be Pareto-better than SSE-Lag and Sauté-SAC on the training layouts. Second, on held-out layouts, success must stay within 10 points of in-distribution while the world-frame ablation drops sharply, with episode cost remaining below budget.
Why not just…?
Why not just run a flat safe agent on agent-centric observations? The UAS work shows this is excellent for reactive avoidance, with over 99% success in open airspace. But SSE reports that flat and weaker hierarchical baselines reach 0% on multi-stage AntDoubleKeyChest. Our hazard-augmented bottleneck mazes are built to require non-reactive routing, so the flat Sauté-SAC and PPO-Lagrangian baselines directly test whether hierarchy is needed. If they match us, the hierarchy is unjustified on these tasks and we will say so.
Why not just add a Lagrangian cost penalty to an existing hierarchical agent? This is the real threat. SSE-Lag may already explore well and meet the budget on average. Our argument is about when and where violations happen. A single multiplier reacts only after violations have accumulated, and nothing in its world-frame critic transfers. The comparison of in-training cumulative cost, together with the held-out-layout test, will settle it. If SSE-Lag matches us on held-out layouts, our contribution reduces to the relative representation, and the budget-slice ablation will show whether allocation adds anything.
Why not just use a classical planner with a safe local tracker? On known static maps this may win, and the A* + Sauté baseline is included for that reason. HiSaRL itself notes that analytical controllers need accurate dynamics models, which an ant quadruped lacks. The planner also cannot account for the local cost of a route, only its length. We expect it to take short but hazard-dense corridors, and the per-hop cost overrun metric will show whether it does.
Why not just pass the full remaining budget to a relative-frame worker? This is our first ablation. Without slices, the worker has no reason to save budget for later hops. It may spend the budget early and leave the manager stranded, a failure we expect to see as a late-episode stall. If that stall does not appear, allocation is unnecessary, and the simpler design should be preferred.
Proposed Experiments
We will use Safety Gymnasium's PointGoal and CarGoal at hazard level 2, plus AntMazeBottleneck and Ant FourRooms augmented with Safety-Gymnasium-style hazard discs placed in corridors. Training uses four hazard layouts per maze. Evaluation uses held-out layouts in three forms: hazards shifted, the maze rotated by 90°, and the maze mirrored. A secondary test swaps the Point body for Car at test time without retraining the manager, echoing RLSF's cross-morphology transfer of cost functions (RLSF).
The metrics are:
- success rate;
- episode cost relative to the budget;
- cumulative cost over all of training, which is the in-training safety measure;
- per-hop calibration, defined as the fraction of hops whose realized cost exceeds $b_k$;
- the gap between in-distribution and held-out success.
Three checks are decisive. First, on held-out layouts, the cost-to-reach critic with agent-centric input should keep its prediction error close to its training error, while the world-frame version and the absolute-position version degrade. If this fails, the transfer premise is wrong. Second, cumulative in-training cost should be lower than both SSE-Lag and flat Sauté-SAC at matched final success. Third, per-hop overrun should stay rare on held-out layouts. If overruns rise, the conservative ensemble quantile is the lever to test, and we will report the return it costs.
References
- HiSaRL: A Hierarchical Framework for Safe Reinforcement Learning
- Safe Reinforcement Learning via Hierarchical Adaptive Chance-Constraint Safeguards
- Safe- to-Explore State Spaces: Ensuring Safe Exploration in Policy Search with Hierarchical Task Optimization
- SAUTE RL: Almost Surely Safe Reinforcement Learning Using State Augmentation
- Constrained Markov Decision Processes via Backward Value Functions
- Strict Subgoal Execution: Reliable Long-Horizon Planning in Hierarchical Reinforcement Learning
- Active Hierarchical Exploration with Stable Subgoal Representation Learning
- Observation Space Matters: Benchmark and Optimization Algorithm
- Bimanual Regrasping for Suture Needles using Reinforcement Learning for Rapid Motion Planning
- Obstacle Avoidance for UAS in Continuous Action Space Using Deep Reinforcement Learning
- Inter-Agent Relative Representations for Multi-Agent Option Discovery
- Safety Gymnasium: A Unified Safe Reinforcement Learning Benchmark
- First Order Constrained Optimization in Policy Space
- Safety through feedback in Constrained RL