An agent that learns safely in one room is usually not safe in the next room. Current safe exploration methods train a model of what could go wrong over the whole observation: a flat vector of sensor readings or pixels. They then fall back to a cautious default policy whenever that model is unsure. Moving the obstacles, adding a third one where there were two, or rotating the arena makes the whole state look new. The agent then either stops making progress because it keeps falling back, or it acts on a model that is confidently wrong. We propose a different structure. The safety model predicts cost one hazard at a time, from the agent's position and velocity relative to that hazard. It also keeps track of its uncertainty separately for each kind of hazard. Total risk is assembled from these per-hazard pieces. In a new layout made of familiar hazard types in unfamiliar numbers and positions, every piece is still familiar. The safety certificate therefore carries over without any new data, and the agent stays cautious only where something truly new appears.
The bet is that most physical safety constraints are local, so whether you collide depends on you and the thing you collide with. If that holds, the data needed to learn safe behavior should grow with the number of hazard types, not with the number of layouts. The same factoring also eases a known failure of safe agents. Because every encounter with any instance of a hazard type trains that type's model, each type sees far more near-miss data than a whole-state model sees for any one configuration.
Core Claim
Consider a safe-exploration agent of the kind introduced by Safe Exploration via Policy Priors (SOOPER). It pairs a learned probabilistic world model with a safe but mediocre default policy, and hands control to that default whenever predicted cost or model uncertainty threatens the budget. Suppose its whole-state model is replaced with a factored model of the form
$$\hat c(s,a) = \sum_{e \in E(s)} g_{k(e)}\big(z_e, a\big), \qquad u(s,a) = \max_{e \in E(s)} u_{k(e)}\big(z_e, a\big).$$
Here $E(s)$ is the set of entities present in state $s$, $k(e)$ is the type of entity $e$, $z_e$ is the agent's state relative to $e$ (offset, relative velocity, heading), $g_k$ is a small ensemble predicting short-horizon cost and relative motion for hazards of type $k$, and $u_k$ is that ensemble's disagreement. The claim is that this agent has three properties. It keeps SOOPER-level training safety in the layouts it was trained on. On held-out layouts with more hazards or rearranged hazards of known types, it incurs near-zero violations with no extra samples and with a fallback rate that does not grow with layout novelty. When a hazard type it has never seen appears, it becomes cautious only near that hazard.
This is a "deictic" representation in the sense of Zero-Shot Transfer with Deictic Object-Oriented Representation in Reinforcement Learning, meaning one anchored to a focal object rather than to absolute coordinates. That paper learns transition rules relative to focal objects and transfers them from a roughly 7,900-state Sokoban level to a roughly one-million-state one. Its exploration rule, however, deliberately visits whatever is unknown, with no notion of cost, and it assumes deterministic, invertible dynamics. We borrow its anchoring idea but apply it to cost and to epistemic uncertainty under stochastic continuous dynamics, and we couple it to a fallback mechanism instead of curiosity-driven exploration.
Why This Gap Is Real
The motivating evidence comes from three directions. First, the spatial generalization literature documents flat policies collapsing under trivial layout changes. HackAtari reports a PPO agent's Boxing score falling from 90.9 to 1.9 under a mechanic change. The Overcooked Generalisation Challenge shows that mirroring or rotating kitchens breaks cooperative policies. MetaDrive needs 100+ procedural training maps to reach about 80% success on unseen ones.
Second, the safe-exploration methods with the strongest training-time guarantees all model the state flatly. SOOPER reports zero violations throughout training, but only within fixed task setups. Guiding Safe Exploration with Weakest Preconditions (SPICE) fits local linear models and filters actions through a small optimization problem, reporting an 89% violation reduction over Constrained Policy Optimization. Its safe region is still expressed in fixed state coordinates. Constrained Markov Decision Processes via Backward Value Functions keeps every intermediate policy within budget by tracking cost accumulated so far, but over fixed grid or joint vectors.
Third, the only relational work that is sample-efficient is unconstrained. Efficient Exploration and Discriminative World Model Learning with an Object-Centric Abstraction (MEAD) predicts whether high-level object-attribute changes succeed. It reports 10x to 100x fewer samples than Dreamer-v3 and PPO, but it has no cost signal at all.
Two pieces of prior work constrain the design. Explainably Safe Reinforcement Learning compresses a 232,000-entry safety shield into a 57-node decision tree. Its authors note that without hand-supplied relational predicates such as "distance to nearest car," the trees split on raw coordinates and lose meaning. This tells us the relational inputs matter. It also tells us not to depend on hand-written predicates beyond entity type and relative pose, which simulators provide. The second constraint is the methodological tension MEAD exposes. Discriminative models are cheap to learn, but certifying safety over several steps needs some forward prediction. We resolve this by keeping forward prediction only where safety needs it: a short horizon of the agent-relative-to-hazard state, which is low-dimensional. Long-horizon task planning can remain discriminative or model-free.
Research Program
The system has three parts.
The first part extracts entities. In Safety-Gymnasium and MetaDrive we read entity types and poses from the simulator. In Atari we use the RAM-based extractor of OCAtari, which reports an average F1 of 93.9% and runs about 50 times faster than vision-based extraction. We treat unsupervised entity discovery as out of scope and say so plainly. The question here is whether the factoring helps when entities are given.
The second part is the factored safety model. For each entity type we train a small ensemble that takes the relative state $z_e$ and the action, and predicts per-step cost contribution and next relative state over a horizon of $H$ steps, typically 10 to 30. The predicted cost of a candidate action sequence is the sum over entities of each type's predicted cost along the rollout. Uncertainty is the maximum disagreement across entities. A maximum is used rather than a sum so that a single unfamiliar hazard is enough to trigger caution, while many familiar hazards do not add up to false alarms. Cost already accumulated is carried as an extra state variable, following the backward value function idea, so that the budget is enforced over the whole trajectory.
The third part is the control loop, which follows SOOPER. A task policy, trained with any reward-maximizing learner, proposes actions. If the factored model's pessimistic cost estimate plus cost already spent would exceed the budget, or if uncertainty exceeds a threshold, control passes to the safe default policy, which brakes or retreats.
We will analyze two things. The first is a sample-complexity argument: if per-type models are accurate on the relative-state distribution, then the certificate's error on a new layout grows with the number of hazards near the agent at once, not with layout novelty. This follows directly from the additive form above. The second is where that argument breaks. Two hazards can jointly create danger, for example a gap too narrow to pass. We test this directly and add an optional second-order term over pairs of nearby entities, used only when the residual error of the first-order model demands it.
Baselines
The naive reference is PPO-Lagrangian on the default flat observations, taken from the SafePO library shipped with Safety Gymnasium. That paper reports that PPO-Lag cut violations by 98% relative to plain PPO in its velocity tasks, at the cost of a 45% drop in reward. PPO-Lag only meets constraints at convergence, so it is expected to violate them on every new layout. It is the "do nothing special" point that shows how large the transfer problem is.
The strongest existing competitor is SOOPER with its flat ensemble world model and the same safe default policy we use. It is the strongest because it is the method in the file that achieves zero training violations and comes with a regret guarantee. If anything matches our training-time safety, it is SOOPER. It reports zero violations in all simulated tasks in its own setups. We found no reported result on Lacuna for SOOPER, or any safe-exploration method, on held-out spatial layouts. That held-out setting is exactly what we will measure. To be fair to SOOPER, we will run it with domain randomization over the same training layouts we use, so that both methods see identical data. Both get the same environment-step budget, the same ensemble size and compute, and the same hyperparameter search of 20 configurations each, selected on training layouts only.
We will also run SPICE as the shielding competitor, using its released dynamics and constraint formulation on the tasks where its linear safe-set specification applies. We include FOCOPS from First Order Constrained Optimization in Policy Space as the strongest pure constrained-optimization learner. It reports 1106.1 reward on HumanoidCircle under a cost limit of 50.
The key ablation keeps the object-centric input but removes the factoring. It is a SOOPER-style ensemble whose world model is a permutation-invariant set network over the same entity list, meaning a network whose output does not depend on the order entities are listed in. It pools all entities before predicting, so it produces a single global uncertainty. This separates "relational input" from "per-hazard-type uncertainty." Our claim is specifically about the second.
Beating the field means the following. On held-out layouts, cumulative violations should be at least an order of magnitude lower than PPO-Lag and FOCOPS, and statistically lower than domain-randomized SOOPER and the pooled set model. Return should be at least as high as SOOPER's, and the fraction of steps spent in fallback should not grow with target-layout novelty.
Why not just…?
Why not just randomize layouts for a flat safe agent? This is the most natural alternative, and it gets a full arm in our experiments: domain-randomized SOOPER. The existing evidence is unfavorable but not conclusive. MetaDrive's best randomized agent still fails about 20% of unseen maps. Overcooked agents trained over many layouts still break under mirroring. Neither result involves a safety budget, though. Our experiment separates two cases. In one, the test layouts stay inside the randomized range. In the other, they go outside it with more hazards than ever seen in training. Randomization should do well in the first case and fail in the second. If it also matches us in the second, our claim is wrong.
Why not just use egocentric lidar, which is already relative? This is a real threat. Safety-Gymnasium's lidar is centered on the agent, so a flat model on lidar already ignores where in the arena the agent stands. Our response is empirical. Lidar bins merge nearby hazards of different types into one reading, saturate as hazard density rises, and hide hazards behind one another. Those are exactly the conditions our count-scaling and mixed-type tests create. We will report the comparison at 1x, 2x, and 4x the training hazard count. If flat-lidar SOOPER stays safe even at 4x, the benefit of the factoring is confined to mixed-type and occluded scenes, and we will narrow the claim to say so.
Why not just run a Lagrangian method on object features? OCAtari shows that PPO on object states performs as well as pixel-based PPO. A cost-constrained PPO on object features is therefore a fair proposal. However, Lagrangian methods only satisfy the constraint at convergence. SOOPER reports that primal-dual baselines violate constraints during early training, which is the regime that matters on a new layout. We include this as PPO-Lag on entity lists. Its violations during the first episodes on each target layout are the deciding number.
Why not just learn one relational world model with pooled uncertainty? This is the ablation described above. If it matches the factored model, then the contribution is relational input alone and per-type uncertainty adds nothing. We would report that result as the finding.
Proposed Experiments
On Safety-Gymnasium we use the Goal, Button, and Push tasks with Point and Car robots. Training runs on layouts with hazard and vase counts at the standard level-1 setting. Evaluation runs on held-out seeds at 1x, 2x, and 4x hazard density, and on a mixed condition that adds a hazard type absent from training.
On MetaDrive we train on 10 procedural maps with traffic. We evaluate on unseen maps containing new block types and higher traffic density. Cost is collisions and leaving the road.
On HackAtari we define cost as collisions in Freeway and evaluate on variants that move or recolor cars. Recoloring is a good test of the factored model, because it relies on entity type rather than appearance.
We report three primary metrics on each target layout: cumulative constraint violations during the first N episodes with no fine-tuning, the fraction of steps under fallback, and normalized return. The secondary metric is the number of training environment steps needed to reach a fixed fraction of asymptotic return.
The decisive check is a single plot per domain: violations and fallback rate against target hazard count. Our method's curves should stay flat. Flat-model and pooled-model curves should rise, either through more violations or through more stalling. A second check introduces a novel hazard type. Here our model should fall back near that hazard and nowhere else, and we will measure this by where fallbacks occur in space.
A third check targets the safety-paradox concern raised in Breaking Safety Paradox with Feasible Dual Policy Iteration: as an agent becomes safe, it stops collecting near-violation data and its cost estimates degrade. We will log, over training, the error of each per-type cost model in near-boundary states and compare it with the flat model's error. Pooling across instances of a type should keep our error lower.
Finally, a narrow-gap scenario, where two hazards are individually safe but jointly blocking, measures how much the first-order factoring loses and whether the second-order pair term recovers it. A large loss here would bound the scope of the claim, and we will report it as such.
References
- Safe Exploration via Policy Priors
- Zero-Shot Transfer with Deictic Object-Oriented Representation in Reinforcement Learning
- HackAtari: Atari Learning Environments for Robust and Continual Reinforcement Learning
- The Overcooked Generalisation Challenge
- MetaDrive: Composing Diverse Driving Scenarios for Generalizable Reinforcement Learning
- Guiding Safe Exploration with Weakest Preconditions
- Constrained Markov Decision Processes via Backward Value Functions
- Efficient Exploration and Discriminative World Model Learning with an Object-Centric Abstraction
- Explainably Safe Reinforcement Learning
- OCAtari: Object-Centric Atari 2600 Reinforcement Learning Environments
- Safety Gymnasium: A Unified Safe Reinforcement Learning Benchmark
- First Order Constrained Optimization in Policy Space
- Breaking Safety Paradox with Feasible Dual Policy Iteration