Spline Control Points as a Certifiably Smooth Action Space for Safe Reinforcement Learning

Reinforcement learning agents for robots usually output a fresh motor command at every timestep. Left alone, those commands jitter. Engineers then add penalty terms to the reward ("subtract λ times the change in action") and spend weeks tuning λ: too small and the motors still chatter, too large and the robot refuses to move. Safe reinforcement learning has removed this kind of weight-tuning for collisions. It treats danger as a separate cost with its own budget rather than folding it into the reward. Smoothness has never received the same treatment, and per-step safe RL methods are known to produce exactly the oscillating commands that damage hardware.

We propose a different action space. The policy does not emit a command. It emits the next few control points of a cubic B-spline, a piecewise polynomial curve that stays inside the convex hull of its control points and whose derivatives are themselves B-splines built from finite differences of those points. Because of these two properties, bounds on velocity, acceleration, jerk, and joint position for the entire continuous-time trajectory become a small set of linear inequalities on the control points. Before execution, a cheap projection enforces those inequalities. Smoothness thus becomes a hard constraint, stated in physical units taken from an actuator datasheet and holding between sensor samples, rather than a reward weight or an expected-cost budget. Environmental hazards such as obstacles and contact stay in a standard constrained optimizer.

Core Claim

A constrained Markov decision process is a reinforcement learning problem in which the agent maximizes reward while keeping the expected cumulative cost below a budget. Our claim is that the right division of labor inside such a problem is the following. Kinematic limits are known in advance, convex, and must hold on every trajectory, so they are enforced exactly through the geometry of the spline. Environmental costs are unknown and learned from experience, so they are handled by an existing constrained policy optimizer.

For a uniform cubic B-spline with control points $p_i$ and knot spacing $\Delta t$, the $k$-th time derivative is a B-spline of degree $3-k$ whose control points are

$$p^{(k)}_i = \frac{1}{\Delta t^{k}} \sum_{j=0}^{k} (-1)^{k-j}\binom{k}{j}\, p_{i+j},$$

and by the convex hull property, $\lVert p^{(k)}_i \rVert_\infty \le b_k$ for all relevant $i$ implies that $|q^{(k)}(t)| \le b_k$ at every instant $t$ in the segment, where $q(t)$ is the commanded joint trajectory and $b_k$ is the physical bound on the $k$-th derivative. Each new policy step therefore reduces to projecting a few new control points onto a polytope. The earlier points are already committed, which preserves continuity. The projection is a small quadratic program in a few dozen variables. We also add a braking condition: the remaining horizon must admit a trajectory that comes to rest within the position limits. This keeps the projection feasible at the next step and is the spline analogue of the "can still stop" reasoning that per-step rate clipping cannot express.

The novelty is not "splines in RL" by itself. It is the use of the convex hull property to turn smoothness and kinematic limits into per-trajectory guarantees inside an online safe-RL loop, so that no smoothness weight is tuned at all.

Relation to Prior Work

Two papers motivate the proposal by documenting the jitter problem and treating it with tuned losses. CAPS adds temporal and spatial smoothness penalties to the policy loss and reports large power savings on a real quadrotor. Grad-CAPS shows that the first-order CAPS penalty over-smooths, and penalizes second differences of actions instead. Both keep weighting hyperparameters that their authors acknowledge must be tuned, and neither has a cost budget for hazards. We replace their weights with bounds that have units, and we place the hazards in a constrained process.

Two papers supply the spline machinery. STORM fits B-splines to sampled control sequences inside a model-predictive controller on a Franka arm. Uniform B-spline replanning for aerial vehicles exploits local support and approximation, so that moving one control point only changes a short window of the trajectory. We reuse local support to make online commitment cheap. Unlike STORM, our system learns a policy and does not need a dynamics model at run time.

MoRe-ERL is the closest existing combination of splines and safety-motivated learning. It learns B-spline residuals on top of an RRT-Connect reference path, but it learns episodically and handles safety through reward penalties. Our policy is closed-loop, needs no reference planner, and uses formal cost budgets.

Several safe-RL papers supply the optimizer and also constrain our design. FOCOPS solves a nonparametric constrained update and then projects it onto the network; we use it, together with Lagrangian PPO from SafePO, as the cost optimizer. Backward value functions already use a per-step linearized safety layer, and the file notes that this layer can produce discontinuous commands. Our projection is the smooth counterpart, and it is exact rather than linearized for the convex kinematic part. BSPG proves that optimal constrained policies sit on the constraint boundary. This makes the chattering concern sharp: a jittery policy held at the boundary crosses it repeatedly. We test whether spline actions reduce such crossings. FDPI relies on deliberate violations and is therefore simulation-only. We do not try to resolve its safety paradox here, but our hard kinematic layer would remain active even under exploratory dual policies.

Research Program

The system has three parts. The policy network maps the observation to a Gaussian over the next $m$ control points, with $m$ between 1 and 3, expressed as offsets from the last committed point. The projection layer solves the quadratic program for velocity, acceleration, jerk, position-limit, and braking constraints, and returns the executed control points. A low-level evaluator samples the spline at the simulator or controller rate, for example 20 Hz for the policy and 500 Hz to 1 kHz for the joints.

For the likelihood-based optimizers, we compute log-probabilities on the pre-projection sample and treat the projection as part of the environment. This is the standard safety-layer convention, and we test its bias against a variant that differentiates through the projection with an implicit quadratic-programming layer.

The analysis has three parts. The first verifies the inter-sample guarantee empirically at 1 kHz. The second measures how often the braking condition binds, which tells us how conservative the layer is. The third studies interaction with the cost optimizer: whether the hard kinematic set changes how fast the Lagrange multiplier settles, and whether boundary-tracking policies chatter less across the cost limit.

Baselines

The naive reference is vanilla Lagrangian PPO with per-step actions and no smoothness treatment, run from the SafePO implementation released with Safety-Gymnasium. That paper reports that on velocity-constrained tasks this method achieved a 98% reduction in safety violations relative to PPO, with a 45% drop in reward. It fixes the safety-reward operating point that we must not degrade.

The strongest existing constrained optimizer in the file is FOCOPS. On Humanoid-Circle under a cost limit of 50, it reports a reward of 1106.1, against 697.5 for TRPO-Lagrangian and 671.0 for CPO (FOCOPS). We run FOCOPS both per-step and with our spline action space, and beating it means matching its reward within seed noise at the same cost limit while satisfying hard kinematic bounds. For smoothness, the strongest method is SAC with Grad-CAPS. On DMControl Walker it reports an action fluctuation of 0.41 ± 0.03, against 0.48 for CAPS and 0.53 for vanilla SAC, and on Reacher it reports a reward of 982.20 ± 11.40 (Grad-CAPS). We combine it with Lagrangian PPO and FOCOPS as "Grad-CAPS + constrained". BSPG and FDPI report no numbers that we could find on Lacuna, so we rerun them only if code is available and report them as reference points.

Four ablations isolate the key ingredients. The first uses spline actions without projection, so smoothness comes from the parameterization alone. The second uses projection on per-step actions without splines: discrete rate and jerk clipping with the same physical bounds. The third uses spline actions with jerk handled as a second Lagrangian cost instead of a hard bound. The fourth varies the number of new control points per step.

For fairness, every method sees the same environments, seeds (10 per configuration), environment-step budget, and network sizes, and receives an identical hyperparameter search budget. Baselines get their search spent on smoothness weights; our method gets its search spent on optimizer learning rates only, since the bounds come from the robot specification.

Why not just…?

Why not just add jerk as a second cost in a Lagrangian method? This is the most natural alternative, and it does remove the hand-tuned weight. However, an expected-cost budget bounds average jerk, not jerk on each trajectory. During training it is violated until the multiplier settles, and it is known to oscillate. The third ablation tests this directly. If it matches our method on maximum per-episode jerk and on training-time kinematic violations, our hard layer is unnecessary for smoothness, and the contribution narrows to inter-sample position guarantees.

Why not just clip the change in each action, or low-pass filter it? This is the real threat, because per-step rate clipping is also a hard projection. It falls short in two ways that we can measure. First, it constrains only sampled commands, not the continuous signal between policy steps when the low-level controller runs faster. Second, it cannot anticipate: a rate-limited command can run into a position limit because braking needed to begin earlier. The second ablation, evaluated at 1 kHz with tight position limits, settles this. If clipping matches us there too, we will report that the value lies in hard projection and not in splines.

Why not just use CAPS or Grad-CAPS on top of a constrained optimizer? These methods reduce average fluctuation, and Grad-CAPS reduces it markedly on Walker. However, they still require weights that their authors note can over-smooth, and they give no bound on any trajectory. The sensitivity sweep below shows whether their tuning cost is real. If a single default weight works across all tasks, this objection to them weakens.

Why not just run the policy at a lower rate with action repeat? Holding an action constant produces step changes at every boundary, so velocity is still discontinuous. Our spline-without-projection ablation already dominates this option on smoothness, and we include action repeat only as a sanity check.

Proposed Experiments

The main suite uses Safety-Gymnasium navigation and velocity-constrained locomotion: Point and Car Goal tasks, the MuJoCo velocity tasks, and Humanoid-Circle as used by FOCOPS. We add DMControl Walker and Reacher, where the cost is zero and only the smoothness comparison with Grad-CAPS applies, and a joint-limited manipulation task from Safety-DexterousHands.

The metrics are episodic return, episodic cost, and cumulative cost during training. Smoothness is measured with the Grad-CAPS fluctuation measure and with maximum and RMS jerk computed from the 1 kHz continuous signal. We also count kinematic-limit violations between policy steps and boundary crossings per episode near the cost limit.

Three checks decide the outcome. The first is that kinematic violations between samples are zero for our method and nonzero for the per-step clipping and Lagrangian-jerk ablations. The second is that return and cost stay within seed noise of per-step FOCOPS and Lagrangian PPO. Losing more than about 10% of reward would mean the braking condition is too conservative, and we would report that. The third is a tuning-sensitivity sweep: across a 1-to-100 range of smoothness weights for the baselines, versus a ±50% perturbation of the physical bounds for our method, we plot the spread of return and maximum jerk. The proposal succeeds only if our method is both at least as safe and much less sensitive.