Reinforcement learning agents on physical machines are often wrapped in a safety filter: a small optimizer that checks each proposed action and, if that action could eventually push the system out of its allowed operating region, replaces it with the nearest action that is still safe. These filters are only as good as their model of the machine. Existing filters assume the model error is either an additive disturbance of known size or a smooth function of the current state that a Gaussian process can learn. Real machines also have hidden dynamics: flexible structural modes, actuator lag, and sensor filters. These carry their own internal state and react to the history of inputs, not just the current one. When such dynamics shift, the consequences are severe. An adaptive controller that ignored them went from a 0.0647 tracking error to outright instability in the experiments of Safety Filter for Robust Disturbance Rejection via Online Optimization.
We propose a safety filter whose model error is a linear fractional representation: the plant is written as a known linear system $M$ in a feedback loop with an unknown operator $\Delta$ (a separate dynamical block that takes a signal out of the plant and feeds a correction back in). The filter's guarantees then depend only on a bound on how much $\Delta$ can amplify signals. The new ingredient is that this bound is not fixed in advance. It is certified from data the filter itself collects, using set-membership identification of $\Delta$'s impulse response. The bound is tightened online, and an exploration bonus steers the agent toward inputs that shrink it fastest. The result is a filter that starts conservative under dynamic uncertainty, provably never violates state constraints, and loosens its own restrictions as it learns how large its blind spot really is.
Core Claim
Take a known nominal plant with full state measurement. The unmodeled part enters as
$$x_{k+1} = A x_k + B u_k + B_p p_k + w_k,\qquad z_k = C_z x_k + D_z u_k,\qquad p = \Delta(z).$$
Here $x_k$ is the state, $u_k$ the input, $w_k$ a disturbance bounded in a known set, and $z_k$ and $p_k$ the signals through which the hidden dynamics $\Delta$ (a causal, stable, linear time-invariant operator) interact with the plant.
We assume a prior envelope on $\Delta$'s impulse response $h$, namely $|h_i| \le c\rho^i$, the kind of decay bound a structural engineer can supply. The past residuals $B_p p_k$ are observable up to the bounded noise $w_k$. The first $L$ taps of $h$ therefore lie in a polytope $\Theta_t$ that shrinks as data arrive. The certified gain bound is
$$\gamma_t = \max_{h \in \Theta_t} \sum_{i<L} |h_i| \;+\; \frac{c\rho^L}{1-\rho}.$$
This is the peak-to-peak (induced $\ell_\infty$) gain of $\Delta$, which equals the $\ell_1$ norm of its impulse response. The maximization is a small linear program for a single-input, single-output block.
Plugging $\gamma_t$ into a robust System Level Synthesis safety filter gives the main result. System Level Synthesis is a method that optimizes the whole closed-loop response from disturbances to states and inputs, rather than a fixed feedback gain. The claim is that the resulting filter keeps $x_k$ in the safe set for every $\Delta$ consistent with the prior and the data. Because $\gamma_t$ is non-increasing, the filter's safe set grows monotonically.
Relation to Prior Work
Three papers supply the machinery, and none does what we propose.
Predictive safety filter using system level synthesis builds a filter around black-box learning controllers with near-maximal safe sets and a 0.031 ms explicit evaluation, but it assumes $(A,B)$ is exactly known.
Robust Closed-loop Model Predictive Control via System Level Synthesis adds a linear fractional uncertainty block to the same framework and enlarges feasible regions relative to tube methods. However, it is a regulator with a fixed, hand-chosen uncertainty bound and offline hyperparameter search, not a filter around an exploring agent.
Safety Filter for Robust Disturbance Rejection via Online Optimization is the closest in spirit: it guards an online optimizer with a small-gain, peak-to-peak bound on $\Delta$. It enforces stability only, not state constraints, its bound $\gamma$ never changes, and it reports the price of that fixed bound (nominal error 0.0647 rising to 0.0876).
Our proposal takes the robust filter structure from the first two papers and the peak-to-peak small-gain argument from the third. Its distinct contribution is a data-certified, shrinking $\gamma_t$ coupled to exploration.
Two lines motivate the learning half. Learning-enhanced robust controller synthesis with rigorous statistical and control-theoretic guarantees shows that tightening a learned uncertainty bound pays off: its performance index fell from 10.5575 to 3.1581 with 50 samples. Its data are passive and offline, and its bound is a static sector bound on a memoryless nonlinearity rather than a gain bound on a dynamic operator.
Safe exploration of nonlinear dynamical systems: A predictive safety filter for reinforcement learning, A predictive safety filter for learning-based control of constrained nonlinear dynamical systems, and Safe Guaranteed Exploration for Non-linear Systems do let exploration loosen the filter as uncertainty shrinks. They model error as a function of the current state and input, and that assumption breaks when the error has memory. This is precisely the regime where our formulation applies.
Research Program
The work has three parts.
The first part is the certified bound. We will build the set-membership estimator for $\Theta_t$ and prove that $\gamma_t$ upper-bounds the true gain whenever the decay prior and the disturbance bound hold. We will also study how fast $\gamma_t$ shrinks as a function of input excitation. The intuition is that frequencies the agent never excites leave the corresponding directions of $\Theta_t$ unconstrained.
The second part is the filter. We will extend the robust System Level Synthesis program with a small-gain constraint on $\Delta$ that uses $\gamma_t$, then solve it online. As a fast mode, we will precompute explicit filters for a grid of $\gamma$ values and switch among them as $\gamma_t$ crosses grid points. This keeps the explicit filter's speed while still benefiting from learning.
The third part is exploration. We will add an intrinsic reward equal to the predicted reduction in $\gamma_t$ from a candidate input sequence, computed by a linear-program sensitivity of the bound. The agent will be trained with an off-the-shelf policy-gradient method.
A key honesty point is that guarantees hold only if the decay prior is valid. We will therefore test deliberately violated priors and report how and when the filter fails.
Baselines
The naive reference is the unfiltered reinforcement learning agent. We will run the same policy-gradient learner with the same reward and training budget and count constraint violations and divergences. On the Lai–Seiler style uncertain plant, the comparable unprotected adaptive scheme is reported as unstable.
The strongest Lagrangian competitor is PPO-Lag from Safety Gymnasium: A Unified Safe Reinforcement Learning Benchmark. It will be run through its SafePO implementation with the same environment and the same number of environment steps, with its cost limit tuned over the same grid size we allow our exploration weight. On velocity tasks it reports a 98% reduction in violations relative to PPO at a 45% reward drop. Beating it means zero training violations at equal or higher final return.
The strongest filter competitor is the known-plant System Level Synthesis filter of Predictive safety filter using system level synthesis. We will run it in two ways. In the first, the hidden-dynamics contribution $B_p p$ is ignored entirely. In the second, which is the key ablation, $B_p p$ is folded into an enlarged additive disturbance set sized from offline worst-case simulation. Both versions get identical horizons and solvers, and we will report safe-set volume, intervention rate, and violations. That paper reports no intervention rate, so this number will be new.
The robust fixed-bound filter, in the style of Robust Closed-loop Model Predictive Control via System Level Synthesis, uses our filter with $\gamma$ frozen at its prior value $\gamma_0$. This isolates the value of learning the bound.
The Gaussian-process filter of A predictive safety filter for learning-based control of constrained nonlinear dynamical systems will be given the same initial data and the same per-step compute budget.
A second ablation removes the exploration bonus, so that $\gamma_t$ shrinks only from whatever data the task reward happens to produce.
Our method beats the field if it matches the zero-violation record of the filters, enlarges the safe set faster than the frozen-bound and no-bonus variants, and achieves higher return than all filters at equal compute.
Why not just…?
Why not just treat the hidden dynamics as a bigger additive disturbance? This is the most serious threat. The term $B_p p$ is bounded only by $\gamma\cdot\max|z|$, and $z$ depends on the state and input the filter itself chooses. A fixed additive set is therefore either sized for the worst state, which makes it very conservative, or sized for typical operation, which makes it invalid near the constraint boundary. If the offline-sized additive variant matches our safe-set volume with zero violations on our benchmarks, the feedback-loop formulation is not worth its complexity, and we will say so. The ablation in the Baselines section is designed to settle exactly this.
Why not just learn the model error with a Gaussian process over states and inputs? A Gaussian process maps the current state and input to the next state. Hidden flexible modes make the true error depend on past inputs, which violates that assumption, and the confidence tube can then be confidently wrong. Feeding a window of past inputs to the process restores memory but inflates the input dimension. The comparison on the flexible-mode benchmarks, matched for compute, will measure whether the Gaussian-process filter's violations or conservatism actually materialize.
Why not just use Lagrangian safe RL, or domain randomization? Both are evaluated only on average. PPO-Lag still leaves residual violations (Safety Gymnasium). RRLS : Robust Reinforcement Learning Suite presents domain randomization as strong on average across its parameter range but without worst-case guarantees. Neither can promise zero violations during training on hardware. We include PPO-Lag in the experiments to quantify the reward cost of our guarantee rather than assume it is small.
Proposed Experiments
The first benchmark is the double integrator of the two System Level Synthesis papers, extended with a first- or second-order actuator lag treated as $\Delta$, with polyhedral state and input constraints and randomized true lag parameters. The second is the rigid-body-with-flexible-dynamics plant of Safety Filter for Robust Disturbance Rejection via Online Optimization, with 100 randomly drawn uncertain plants and a position constraint added. On that plant the reported reference is a filtered tracking error of 0.0920 against instability without the filter. We will report the same standard-deviation metric alongside violations. The third is a linearized inverted pendulum with unmodeled actuator dynamics, following the angle-constraint setup of Safe exploration of nonlinear dynamical systems: A predictive safety filter for reinforcement learning. As an out-of-theory stress test, we will use the InvertedPendulum task from SuReLI/RRLS with action-perturbation wrappers. There, linearization error breaks the guarantee, and we expect to observe violations and will report them.
Metrics are cumulative training violations, final return, safe-set volume over time, the trajectory of $\gamma_t$ against the true gain, intervention rate, and per-step filter time against the 0.031 ms and 65.7 ms reference points.
The decisive checks are three. The learned-bound filter must achieve zero violations across all random plants while its safe set grows. It must clearly outperform the additive-disturbance ablation, either in safe-set size or in violations at matched size. And the exploration bonus must shrink $\gamma_t$ measurably faster than reward-only exploration without costing final return. If the first check holds and the second fails, the paper's honest conclusion is that a well-sized additive set suffices for these plants.
References
- Safety filter for robust disturbance rejection via online optimization
- Predictive safety filter using system level synthesis
- Robust Closed-loop Model Predictive Control via System Level Synthesis
- Learning-enhanced robust controller synthesis with rigorous statistical and control-theoretic guarantees
- Safe exploration of nonlinear dynamical systems: A predictive safety filter for reinforcement learning
- A predictive safety filter for learning-based control of constrained nonlinear dynamical systems
- Safe Guaranteed Exploration for Nonlinear Systems
- Safety Gymnasium: A Unified Safe Reinforcement Learning Benchmark
- Rrls: Robust reinforcement learning suite