A robot that learns in simulation and then explores on real hardware faces two problems at once. It must not crash while learning, and its real sensors are noisier than the simulated ones it trained on. Many physical systems also behave differently in different regimes: sliding freely versus touching a wall, a pendulum swinging versus hitting a joint limit. A switching linear dynamical system handles this by approximating nonlinear physics with a small library of linear models plus a rule for which one is active. The regime boundaries in such a model often sit right on top of the safety boundaries. Noisy sensors make it hardest to tell which regime the system is in exactly where a wrong guess causes a crash.
We propose a safety filter that never commits to one regime. It keeps a probability over which linear model is active, tracks how uncertain the continuous state is under each one, and allows only actions that are safe under every regime the sensors cannot yet rule out. Because each regime is linear, this "safe under all plausible regimes" requirement reduces to a small set of linear inequalities on the action, so it can be enforced every control step. It also lets the policy sample directly inside the safe action set instead of being overridden by a projection. The sim-to-real sensor gap then becomes a low-dimensional problem. The dynamics matrices learned in simulation stay fixed, and only the measurement-noise covariance is estimated online. That estimate starts pessimistic and tightens only as real data justify it.
Core Claim
The proposal claims that sensor noise is dangerous in hybrid systems mainly because the controller misidentifies the regime near boundaries, not because the state estimate is slightly off. Two things follow if this is right. A filter that uses the single most likely regime will concentrate its safety violations at regime transitions under unseen noise, while a filter that enforces constraints across the regime posterior will not. Adapting only the measurement covariance on the real system should also recover most of the task return lost to an overly conservative fixed margin, with no increase in violations during that adaptation.
The closest existing work each supplies one piece and omits the others. Hybrid Recurrent Models Support Emergent Descriptions for Hierarchical Planning and Control learns a recurrent switching linear model (rSLDS) and plans over its discovered regimes with low-level LQR controllers. It assumes clean states, has no safety constraints, and names control-theoretic safety as future work. Filter-Based Abstractions for Safe Planning of Partially Observable Dynamical Systems supplies the technique we reuse: Kalman filtering with obstacles inflated by an estimation-error buffer. It handles only a single linear time-invariant system and grids the state space offline, producing 74,000 states for a 6D system. We extend the buffer to a mixture over regimes and apply it online without gridding. Provably Safe Sim-to-Real Transfer motivates confining real-world learning to where simulation is wrong. It is tabular and assumes separable shifts that continuous sensor noise violates. We keep its principle, but the "mismatch" becomes a covariance matrix rather than a set of cells.
Multiple-model filtering is a classical estimation tool. The new part is coupling it with learned rSLDS regimes, per-regime barrier constraints, and safe online adaptation to an unknown sensor noise level.
Research Program
The model is an rSLDS, a switching linear model whose regime switches depend on the current state. It is fit on clean simulator rollouts following Recurrent switching linear dynamical systems. Each regime $k$ has matrices $A_k, B_k$ and process noise $Q_k$. $Q_k$ is inflated by the simulator residuals of the fit so that model error is counted as noise.
At run time a bank of per-regime Kalman filters produces three quantities: a regime posterior $p_k = p(z_t = k \mid y_{1:t})$ from observations $y_{1:t}$, a state estimate $\hat x_k$, and a covariance $P_k$ for each regime. The safe set is a polytope of half-spaces $h(x) = c^\top x + d \ge 0$. For every regime with $p_k \ge \delta$, the control $u$ must satisfy a discrete-time barrier condition tightened by that regime's predicted uncertainty:
$$c^\top (A_k \hat x_k + B_k u) + d \;\ge\; (1-\alpha)\,(c^\top \hat x_k + d) \;+\; \kappa \sqrt{c^\top (A_k P_k A_k^\top + Q_k)\, c}.$$
Here $\alpha \in (0,1]$ sets how fast the system may approach the boundary. $\kappa$ is a Gaussian quantile that fixes the per-step violation probability, and $\delta$ is the posterior mass below which a regime is treated as ruled out. Each inequality is linear in $u$, so their intersection is a convex polytope of safe actions.
We do not project a policy's action onto this polytope. Sampling-based Safe Reinforcement Learning for Nonlinear Dynamical Systems shows that such projections distort the gradient signal the policy learns from and cause it to stall at obstacles. Following that paper, the policy is a Beta distribution supported on an inner box of the polytope. Rival safe-set methods such as Enforcing Hard Constraints with Soft Barriers are limited in two ways. They need smooth dynamics and explicitly fail at impacts, and training takes 8 to 24 hours. The per-regime linear structure is what lets our approach handle discontinuities and run cheaply.
For the sim-to-real sensor gap, the measurement covariance $R$ starts at a deliberately inflated prior. It is updated from the filter's innovation sequence, the gap between predicted and received observations, using covariance matching with a one-sided confidence bound. The bound ensures $R$ shrinks only when evidence supports it. Larger $R$ widens every $P_k$ and flattens the regime posterior, so early real-world exploration is automatically conservative near boundaries.
We test the layer under two learners. One is the Safe-RPG-style policy. The other is the hierarchical agent of Collis et al., in which the high-level planner proposes regime sub-goals and the LQR output is filtered. This second host tests whether the safety layer can be added to an existing rSLDS planner.
The guarantee we will state is conditional. If the rSLDS with inflated $Q_k$ covers the true dynamics and the $R$ bound holds, each step violates a constraint with probability at most the $\kappa$-implied level, summed over the horizon. We will measure empirically how often these premises fail.
Baselines
The naive reference is the same rSLDS policy with no safety layer, trained on clean simulation and deployed under noise. It shows how much the sensor gap alone costs in violations.
The strongest model-free competitor is PPO-Lagrangian trained with sensor-noise domain randomization, run from SafePO in Safety Gymnasium. Randomizing noise during training is the standard practical fix. On 4D Cartpole with clean states, Enforcing Hard Constraints with Soft Barriers reports an empirical safe rate of 89.3% for PPO-Lagrangian and 84.2% for FOCOPS, against 100% for its own soft-barrier method. That 100% is the strongest reported number on the benchmark we share. We will run the soft-barrier method itself under our noise protocol and count wall-clock time.
The strongest model-based competitor under partial observability is PIGDreamer. It reports "near-zero" cost on Safety-Gymnasium PointGoal and CarGoal but no per-task numbers, so no exact reported figure is available on Lacuna. We will rerun it with its privileged simulator states during training and our noisy observations at deployment.
Safe-RPG with a single global linear model plus a Kalman filter is the single-regime ablation. It is the natural continuous analogue of the Filter-Based Abstractions approach. A projection-filter variant, a CBF quadratic program applied to an unconstrained policy, tests the Safe-RPG argument inside our setting.
Two ablations isolate the key ingredient. The first commits to the single most likely regime. The second uses our full layer with $R$ fixed at the pessimistic prior, which isolates the benefit of adapting $R$.
All methods get the same number of simulator steps, the same deployment episodes, and the same hyperparameter search budget (20 configurations each, 5 seeds). They are tuned on the same noise level, which is excluded from test.
Beating the field means two things together. The first is violations during noisy real-phase learning below PPO-Lagrangian with domain randomization and below the MAP-regime ablation, with non-overlapping seed intervals. The second is task return within the range of PIGDreamer, and above the fixed-$R$ layer, at unseen noise levels.
Why not just…?
Why not just randomize sensor noise in simulation and train a constrained policy? This is the serious threat for final return, and it will get a full tuning budget. Its weakness is structural. Expected-cost constraints permit trading an occasional crash against many safe steps. The Cartpole numbers above show this gap even without noise, and Robust Gymnasium finds that CRPO and PCRPO both lose substantial safety under observation noise. Randomization also only covers the noise range it was given. The decisive test is noise levels outside the training range. If domain randomization matches our violations there, the mode-belief machinery is unnecessary and we will report that.
Why not just use one linear model and a Kalman filter with an inflated buffer? This is the ablation that directly tests the core claim. If a single linearization with a larger margin matches our violations without losing return, then regime structure is not what matters. We expect it to fail in one of two ways near walls and joint limits, because one linear model cannot represent both sides of a contact. A small buffer means crashes. A buffer large enough to cover the contact mismatch leaves the agent "safe but useless," the failure described by Quanyi Li et al..
Why not just use a fixed worst-case safety margin and skip adapting to the sensor noise? It is safe by construction, so the question is purely one of conservatism. We will sweep fixed margins and plot return against violations. The claim holds only if online $R$ adaptation reaches a point on that curve the fixed margins cannot reach. If some fixed margin sits on the same frontier, adapting $R$ adds nothing, and the contribution narrows to the mode-mixture constraint.
Why not just a learned black-box safety critic or world model? Sim-to-Lab-to-Real shows such shielding works on hardware, reaching an 86.7% real safety rate. PIGDreamer is strong in simulation but did not help on PointButton1, which has moving objects. These critics give no account of how sensor noise enters their safety margin, so they cannot tighten when noise grows. Running them at unseen noise levels shows whether that matters in practice.
Proposed Experiments
Environments were chosen so that regime boundaries coincide with constraints. They are Continuous Mountain Car with a constraint at the left wall, the regime-rich task used by Collis et al., who report no numeric results; the 4D Cartpole with track limits from the soft-barrier suite; the hard-constrained inverted pendulum and quadcopter from Safe-RPG, which reports a 100% training safe rate on clean states; and Safety-Gymnasium PointGoal and CarGoal.
The "real" system is the simulator with two changes. Robust Gymnasium observation disruptors inject Gaussian noise, slow bias drift, and dropout, and dynamics are perturbed through SuReLI/RRLS. Noise levels are split into train and unseen. Each method learns in clean simulation and then continues learning for a fixed budget in the disrupted system. We record violations per episode during this phase, final return, success rate, compute per step, and wall-clock training time.
Three checks decide the claim. First, we locate every violation relative to the learned regime boundaries. The core claim predicts that the MAP-regime ablation's violations cluster within one step of a regime switch and ours do not. Second, we test whether the filters remain calibrated on the disrupted system, meaning normalized innovations stay consistent with the model, as $R$ adapts. This is exactly the premise our conditional guarantee rests on, and we will report how often it breaks. Third, we compare ours, the projection variant, and the constrained-support policy on tasks where an obstacle lies on the goal path. This settles, in our setting, the disagreement between Safe-RPG's argument against projection and Sim-to-Lab-to-Real's success with shielding.
If the violations of the single-regime or domain-randomized baselines are not concentrated at regime boundaries, the central hypothesis is wrong, and we will say so.
References
- Hybrid Recurrent Models Support Emergent Descriptions for Hierarchical Planning and Control
- Filter-Based Abstractions for Safe Planning of Partially Observable Dynamical Systems.
- Provably Safe Sim-to-Real Transfer
- Recurrent switching linear dynamical systems
- Sampling-based Safe Reinforcement Learning for Nonlinear Dynamical Systems
- Enforcing Hard Constraints with Soft Barriers: Safe Reinforcement Learning in Unknown Stochastic Environments
- Safety Gymnasium: A Unified Safe Reinforcement Learning Benchmark
- PIGDreamer: Privileged Information Guided World Models for Safe Partially Observable Reinforcement Learning
- Robust Gymnasium: A Unified Modular Benchmark for Robust Reinforcement Learning
- MetaDrive: Composing Diverse Driving Scenarios for Generalizable Reinforcement Learning
- Sim-to-Lab-to-Real: Safe RL with Shielding and Generalization Guarantees