A common way to keep a continuous-control agent safe during learning is a safety layer. This is a small module that takes the action the policy proposes and moves it to the nearest action that a learned constraint model predicts is safe. The best-known version handles one constraint at a time. Real robots, though, face several at once: a speed limit, a joint limit, an obstacle. Where these meet, the safe region of actions is a polytope, a shape with flat faces and sharp corners. A reward-seeking agent pushed against such a region tends to park at a corner. It then chooses nearly the same action step after step, and that action is right on the boundary. Two things go wrong. The agent stops collecting the varied boundary data it would need to learn where the constraints actually lie, and the projection distorts its gradients.
We propose to deliberately round off the corners of the estimated safe action set before projecting onto it. The rounding radius is tied to how much the agent still does not know about the constraints. The bet comes from bandit theory, which shows that a curved action set makes greedy, reward-maximizing choices vary smoothly. That variation gives informative data as a free byproduct. If the same holds here, the agent can learn its constraints from safe, near-boundary actions rather than from crashes. Rounding inward never enlarges the safe set, so the zero-violation property of the safety layer is kept. The cost is a controllable loss of reward, roughly proportional to the radius, and we expect it to buy faster learning of the constraint model and fewer samples to reach a good policy.
Core Claim
The claim is that the "safety tax" in sample complexity under multiple constraints comes largely from the sharpness of the safe set. If so, it can be reduced by shrinking the set to a curved inner approximation whose curvature is scheduled against the estimation error of the constraint model.
Three lines of prior work motivate this. First, Hutchinson et al. study safe bandit optimization. They define a "sharpness" measure of the constraint set, prove $\widetilde{O}(T^{2/3})$ regret against the unconstrained $\widetilde{O}(\sqrt{T})$ rate, and show empirically that star-shaped and diamond-shaped sets incur more regret than spheres. Second, Banerjee et al. prove that on action sets with smooth, curved boundaries, any low-regret linear bandit algorithm makes the smallest eigenvalue of its design matrix grow as $\Omega(\sqrt{n})$. Informally, the data it collects keeps covering every direction, without deliberate exploration. Third, Abeille et al. show that randomized exploration behaves well on strongly convex sets but needs artificial inflation on flat ones.
None of this work has been brought into the safe setting. Hutchinson et al. take the geometry of the constraint set as given and pay for it. BallRL proves polynomial sample complexity for linear RL when each state's action set is a ball, but it has no safety constraints that could slice that ball into a sharp region. Our proposal chooses the geometry and measures what the choice buys.
The proposal also differs from the existing ways of handling projection. The Dalal et al. safety layer fits a linear cost model $c(s,a)\approx g(s)^\top a + d(s)$ and projects analytically onto one half-space, assuming at most one constraint is active. That assumption is exactly what fails at corners. Gros et al. derive gradient corrections for projecting onto a known safe set. They show that naive projection piles probability mass onto the boundary and biases policy gradients. They assume the set is known and say nothing about how fast it is learned. We reuse their score-function correction, and we additionally ask how geometry governs the learning of the set itself.
Research Program
The construction is simple. Suppose the current estimated safe polytope at state $s$ is $P(s)=\{a: g_i(s)^\top a + d_i(s) + \beta_i(s) \le 0,\ i=1..m\}$, where $\beta_i$ is a confidence margin from the constraint model. We replace it with its morphological opening of radius $r$. This means first eroding the polytope inward by $r$, then taking the union of all radius-$r$ balls whose centres lie in the eroded set:
$$P_r(s) = \{a: g_i^\top a + d_i + \beta_i + r\|g_i\| \le 0\ \forall i\} \oplus B_r .$$
Here $\oplus B_r$ denotes adding a Euclidean ball of radius $r$. The result $P_r(s)$ lies inside $P(s)$, so the method is at least as safe as the unrounded layer. Its corners and edges are replaced by spherical caps of curvature $1/r$, and its faces stay flat. For a generic linear objective the maximizer lands on a cap rather than on a face, which is all the bandit curvature arguments need.
Projection onto $P_r$ stays cheap. One small quadratic program projects the proposal $a$ onto the eroded polytope, giving a point $p$. Then
$$a^* = p + \min\!\left(1, \tfrac{r}{\|a-p\|}\right)(a-p).$$
When only one constraint is active, this reduces to the Dalal layer with a shifted offset.
The research question is how to set $r$. A fixed $r$ costs reward on the order of $r$ times the reward slope at every step. We propose an uncertainty-coupled schedule in which $r_t(s)$ shrinks as the smallest eigenvalue of the constraint model's design matrix, restricted to the currently active constraints, grows. Large uncertainty buys strong curvature, and therefore diverse boundary data. As the constraints become known, the set sharpens back toward the true polytope.
The theoretical work has two parts. The first is a bandit analysis in Hutchinson et al.'s setting, asking whether such a schedule improves on $T^{2/3}$ for polytopic safe sets. The trade-off between the smoothing bias and the eigenvalue growth on caps of radius $r$ that Banerjee et al.'s argument would give is what decides the exponent. We state $\sqrt{T}$ as the target, not a promise. The second part is a linear-MDP extension in the style of DiffR-BallRL, whose state-dependent ball radii fit $P_r(s)$ naturally.
The empirical work tests whether the mechanism survives deep RL. We run SAC- and PPO-based agents in which the constraint model is a learned network trained on boundary data. The policy gradient is corrected as in Gros et al., with the score evaluated on the unprojected proposal.
Baselines
The naive reference is PPO-Lagrangian with no action-space layer. The Safety-Gymnasium paper reports that on velocity tasks it cut safety violations by 98% relative to PPO, with a 45% drop in reward. This gives the scale of reward loss that an "accept violations, then correct" approach pays. We run it from the Safety-Gymnasium reference implementation, using the same networks and environment-step budget as our method and its default hyperparameter sweep.
The most direct competitor is the Dalal et al. safety layer, which reported zero accumulated violations on Ball-1D, Ball-3D, Spaceship-Corridor and Spaceship-Arena. No reported numeric return or convergence speed was found on Lacuna for those domains, so we rerun it ourselves.
Because the Dalal layer assumes a single active constraint, the stronger and fairer competitor is an exact multi-constraint projection. This is a quadratic program over all linearized constraints, using the same learned cost model and confidence margins, with Gros et al.'s gradient correction. This baseline differs from our method only in the absence of rounding, so it is also the central ablation that isolates curvature.
For sample efficiency we compare against ESPO. It adapts batch size to the angle between reward and cost gradients in parameter space, and it reports 25–29% fewer samples, including 5.7M versus 8M samples on SafetyReacher-v4. For boundary data collection we compare against FDPI, which deliberately gathers violation samples and keeps roughly ten times more violation data in its buffer than its baseline. No reported numeric return or cost was found on Lacuna for it.
All methods get the same environment-step budget, the same network sizes, and an equal hyperparameter-search budget of twenty configurations per method per task, with ten seeds each.
Beating the baselines would mean the following. First, matching the exact-projection baseline's training-time violations while reaching its final return in at least 25% fewer environment steps, which is the margin ESPO claims. Second, achieving faster growth of the constraint model's design-matrix eigenvalue than exact projection on the same tasks.
Why not just…?
Why not just solve the exact multi-constraint projection and apply the Gros et al. correction? This fixes the gradient bias, but it leaves the agent parked at a vertex. Successive projected actions coincide, so the constraint model sees almost no variation in the directions that matter. Abeille et al. and Banerjee et al. both tie that degeneracy to flat geometry. This baseline is our main ablation. If it matches our method on constraint-model eigenvalue growth and on sample efficiency, the curvature hypothesis is wrong and we will say so.
Why not just add exploration noise or entropy to move off the corner? Near a corner, noise projected onto a polytope mostly collapses back onto the same vertex or face. This is the boundary mass pile-up that Gros et al. describe, so added variance does not turn into varied boundary actions. We test this directly with an entropy-boosted exact-projection baseline whose action variance before projection is matched to ours.
Why not just use an interior log-barrier in the action objective? This is a real threat. Maximizing a linear reward plus a log-barrier over a polytope also yields a maximizer that moves smoothly with the parameters, much like the "central path" of interior-point methods. A barrier is arguably another way to curve the set. Our position is that the ingredient that matters is coupling the smoothing scale to constraint uncertainty, not the particular smoothing operator. We therefore include both a fixed-weight barrier and an uncertainty-coupled barrier. If the coupled barrier performs as well as the coupled rounding, the contribution is the schedule. The rounding then stays only as the cheaper, closed-form projection.
Why not just collect violation data on purpose, as FDPI does? FDPI breaks the "safety paradox", where safer policies see fewer violations and so learn less about the boundary, but it does so by spending violations. Our method tries to obtain boundary information without them. The two are complementary. The comparison that matters is constraint-model accuracy per violation incurred, which we report for both.
Proposed Experiments
The first experiment uses Hutchinson et al.'s 3D synthetic safe bandit with sphere, diamond and star constraint sets. We measure cumulative regret, violations, and the smallest eigenvalue of the constraint-estimate design matrix over $T$ up to $10^6$, comparing fixed $r$, coupled $r$, and no rounding. Hutchinson et al. report no numeric empirical regret, so we reproduce their algorithm as the reference. The decisive check is whether the regret exponent fitted on the diamond and star sets drops below 2/3 under coupled rounding while violations stay at zero.
The second experiment modifies Dalal et al.'s Ball-3D and Spaceship-Arena tasks. Each task gets a second and third simultaneous constraint, a speed cap and a wall, so that corners are actually reached. We report accumulated training violations, steps to 90% of final return, and the fraction of steps on which two or more constraints are active.
The third experiment moves to Safety-Gymnasium, using SafetyPointGoal1, velocity-constrained locomotion with box action limits, and SafetyReacher-v4 for direct comparison with ESPO's 5.7M-sample figure. We report normalized return, episodic cost, and environment steps to threshold.
The decisive mechanism check is a known-constraint control. We run every method with the true constraint model given, so that nothing is learned about the safe set. If rounding still helps there, the gain comes from easier optimization rather than from information about the boundary, and the central claim must be narrowed. If rounding helps only when the constraint model is learned, and the help tracks eigenvalue growth on the active constraints, the information argument is supported.
References
- The Impact of the Geometric Properties of the Constraint Set in Safe Optimization with Bandit Feedback
- Exploration in Linear Bandits with Rich Action Sets and its Implications for Inference
- When and why randomised exploration works (in linear bandits)
- Linear Reinforcement Learning with Ball Structure Action Space
- Safe Exploration in Continuous Action Spaces
- Safe Reinforcement Learning via Projection on a Safe Set: How to Achieve Optimality?
- Safety Gymnasium: A Unified Safe Reinforcement Learning Benchmark
- Enhancing Efficiency of Safe Reinforcement Learning via Sample Manipulation
- Breaking Safety Paradox with Feasible Dual Policy Iteration