A controller learned in simulation meets two surprises when it runs on a real robot. The model is wrong in some places and nearly right in others, and the safety constraints, such as "stay 20 cm from that wall," were tuned against the wrong model. The usual fix is to pad every constraint by the same fixed amount. That buys safety everywhere at the price of sluggishness everywhere. We propose a controller that tightens its constraints only where, and only while, the real robot disagrees with the simulator. It then relaxes them again once the disagreement is small and the robot is keeping its margins.
The controller belongs to a family built on a state-dependent coefficient factorization. This rewrites nonlinear dynamics $\dot{x}=f(x,u)$ exactly as $\dot{x}=A(x)x+B(x)u$, so a linear-quadratic regulator gain can be recomputed at every state by solving a Riccati equation. Safety enters through "barrier states," extra state variables that blow up as the robot nears a constraint boundary. The idea is to treat the Riccati cost weight on each barrier state as a Lagrange multiplier. This multiplier is updated online by a damped feedback rule, and it is driven by two signals: the measured constraint margin, and a running estimate of how badly the learned model mispredicts the real dynamics near the current state.
Core Claim
Three pieces of prior work each supply part of this idea. Learning Control-Oriented Dynamical Structure from Data learns $A(x)$ and $B(x)$ with neural networks (SD-LQR). It tracks aggressive PVTOL trajectories from 50–100 samples, with errors an order of magnitude below LQR on learned models, but it has no inequality constraints. Barrier-Riccati Synthesis for Nonlinear Safe Control with Expanded Region of Attraction (BaS-SDRE) adds barrier states to a hand-derived factorization. Its barrier weights are fixed, and it assumes the model is exact. Provably Safe Sim-to-Real Transfer shows, in a tabular setting, that safe real-world learning only needs to pay for the region $B$ where simulator and reality disagree.
The obvious combination is BaS-SDRE running on a learned SD-LQR model. It has not been done, but it inherits the flaw we target: one static barrier weight must cover both the regions where the model is accurate and the regions where it is not. Our claim is narrower and testable. A barrier weight that is scheduled by the model's estimated error and corrected by feedback on the constraint margin reaches the same real-world violation rate as the best uniformly padded constraint, while moving measurably faster and more aggressively. If a uniform pad tuned with the same budget lands on the same safety–agility trade-off curve, the claim is false.
Mechanism
The augmented state is $X=[x^\top, z^\top]^\top$, where $z_j$ is the barrier state for constraint $h_j(x)\ge 0$. It is factorized into $A(X)X+B(X)u$ using the mean-value construction from BaS-SDRE, applied to the learned $A_\theta, B_\theta$. At each control step we solve the algebraic Riccati equation with the state cost
$$Q(X_k)=\mathrm{diag}\big(Q_x,\ \lambda_{1,k}q_z,\ \dots,\ \lambda_{m,k}q_z\big),$$
where $Q_x$ is the tracking weight, $q_z$ is a base barrier weight, and $\lambda_{j,k}\ge 0$ is the adaptive weight for constraint $j$ at step $k$.
The weight is updated as
$$\lambda_{j,k+1}=\Big[\lambda_0+\kappa\,\hat r(x_k)+K_p e_{j,k}+K_i\textstyle\sum_{i\le k}e_{j,i}+K_d(e_{j,k}-e_{j,k-1})\Big]_+ .$$
Here $e_{j,k}=m^\star-h_j(x_k)$ is the shortfall against a desired margin $m^\star$. The term $\hat r(x)$ is an upper confidence estimate of the one-step prediction residual $\|\dot x_{\text{meas}}-A_\theta(x)x-B_\theta(x)u\|$, kept in a small kernel regressor over recently visited states. The gain $\kappa$ converts residual into extra weight, and $[\cdot]_+$ clips at zero.
The integral and derivative terms follow the finding in Safety Gymnasium that plain dual ascent on a Lagrange multiplier oscillates between reckless and frozen behavior, and that PID damping (CPPO-PID) removes the oscillation. The residual term is a continuous, soft version of the mismatch set in Provably Safe Sim-to-Real Transfer. It does not rely on that paper's separability assumption, and it gives up that paper's guarantee.
Two choices separate this from the optimization-side weighting literature. The first is where the weight lives. The error-ratio rule of LADMPSAP and the global penalty increments in Self-Supervised Primal-Dual Learning adjust weights across solver iterations. Our weight adapts across physical time, and it acts on the cost of a feedback law rather than on an optimizer's objective. The second is how constraints are handled inside the control step. Constrained DDP adapts multipliers inside a trajectory optimizer built on local Taylor expansions. We keep the single-shot global Riccati gain and move all adaptation into $Q$. That keeps the per-step cost to one Riccati solve, which we warm-start from the previous step's solution.
Research Program
The work proceeds in four stages. First, we train $A_\theta, B_\theta$ with the SD-LQR consistency loss on simulator data only. Second, we build the barrier-augmented factorization for circular obstacles and wall and gate constraints. Third, we implement the scheduled-weight Riccati controller, with a fallback to the last stabilizing gain whenever the pointwise pair $(A,B)$ is numerically unstabilizable. Fourth, we study transfer in two tiers.
The sim-to-sim tier trains on a nominal model and deploys into perturbed "real" simulators. Following What Matters in Learning A Zero-Shot Sim-to-Real RL Policy for Quadrotor Control?, the perturbations target the terms that actually break transfer: thrust coefficient, motor latency, and drag. We also add state-dependent gusts in the −3 to 3 m/s² range used by QuaDUE-CCM. The simulators are PVTOL and a planar quadrotor, plus 3D Crazyflie models in OmniDrones, whose throughput also makes large tuning budgets for the baselines cheap. The hardware tier flies a Crazyflie with motion-capture state estimates.
Motion capture is a deliberate concession. The Power of Input found that full-state inputs hurt the transfer of neural policies, but a Riccati controller needs the full state. We treat this as a scope limit, not something we solve here.
The analysis tracks each $\lambda_j$ against the true model error along each trajectory. The mechanism is only credible if the weight actually rises where the residual is high and falls elsewhere.
Baselines
The naive reference points are SD-LQR with no constraint handling and fixed-gain LQR. They mark how much the constraints cost and how often an unconstrained controller would collide under the gap. We run them on the same learned model and the same trajectories.
The strongest structured competitor is BaS-SDRE with a static barrier weight, run on the same learned model with a padded margin. Both the weight and the pad are tuned by Bayesian optimization, with the same number of simulator evaluations as our gains ($K_p, K_i, K_d, \kappa$). Tuning uses the same randomized simulators, with randomization restricted as SimpleFlight recommends. This is the competitor the claim must beat. Barrier-Riccati Synthesis reports only qualitative outcomes: BaS-LQR fails in dense clutter and BaS-SDRE succeeds. So there is no reported scalar result on Lacuna for BaS-SDRE collision rates, and we will establish it ourselves. We beat it if, at a violation rate no higher than its own, we achieve at least 15% faster task completion or lower mean tracking error, with non-overlapping confidence intervals across seeds and flights. The sharper form of the test is Pareto dominance over the entire curve that the static pad traces as it varies.
A second strong competitor is BaS-SDRE on a residual-corrected model. It uses the same kernel residual estimator, but adds the estimate to the dynamics instead of to the barrier weight. This isolates whether adapting the weight beats simply fixing the model.
The model-free competitor is a PPO policy with a PID-controlled Lagrange multiplier, as in Safety Gymnasium. It is trained with SimpleFlight's recipe of system identification plus selective randomization, a minimal observation set, and a smoothness penalty, and it is given an obstacle cost. For unconstrained tracking, the reference numbers come from The Power of Input: 3.9 cm error on the 3D Eight-Shape for the best learned input configuration, 4.9 cm for tuned PID, and survival up to 1.21 m/s against 0.73 m/s for privileged inputs. SimpleFlight reports only a relative reduction of more than 50% in mean Euclidean distance, not an absolute figure. On the unconstrained Eight-Shape we must stay within the 3.9 cm range. Otherwise any safety gains could simply be the result of a worse tracker.
Three ablations isolate the key ingredient. The first uses the PID term only, with $\kappa=0$. The second uses the residual term only, with no feedback. The third applies the same PID rule to the margin $m^\star$ instead of to the Riccati weight.
Why not just…?
Why not just pad every constraint by a fixed safety margin tuned under domain randomization? This is the real threat, and the whole proposal hinges on it. A uniform pad is simple and needs no online estimator. SimpleFlight's result that selective randomization transfers well suggests a well-tuned pad could be enough. Our argument is structural. The model error is concentrated in specific regimes, such as high speed, near-saturation thrust, and gusty regions. A single pad large enough for the worst regime slows the robot everywhere else, which is the over-conservatism of robust policies noted in the sim-to-real literature. The Pareto comparison above settles the question directly. If the static pad's curve matches ours, we report that the adaptation is unnecessary.
Why not just correct the model online and keep the barrier weight static? Adaptive modeling is the natural control-theory answer. For slowly varying, learnable errors like a thrust coefficient offset, we expect it to do as well as our method. It should fall short during the first seconds before the residual has been learned, and against stochastic gust forces that no deterministic correction captures. QuaDUE-CCM found such tail disturbances decisive: GP-MPC reached 0% success under extreme disturbance, against 84.1% for its distributional method. The residual-corrected baseline, evaluated separately on constant-offset and gust perturbations, shows which regime favors which approach.
Why not just train a constrained RL policy with a PID Lagrange multiplier and selective randomization? That recipe transfers zero-shot on quadrotors and needs no model. However, its multiplier is fixed after training and cannot react to a mismatch it never saw. The only violation statistic available from the file is the 68.5% unsafe-iteration rate of unconstrained exploration in a 5×5 gridworld, from Provably Safe Sim-to-Real Transfer. That gridworld is not comparable to our tasks. We therefore measure the PPO-PID policy's violation rate under the out-of-randomization perturbations in our sim-to-sim tier. The comparison is also only fair if the policy's own tracking reaches the 3.9 cm reference.
Why not just put a control barrier function filter on top of SD-LQR? A barrier-function filter solves a small optimization at each step to minimally modify a nominal command so that a safety condition keeps holding. It is a reasonable alternative, but the research file contains no paper or reported result for it, so we cannot cite evidence either way. A filter still needs a margin or class-function gain, which raises the same static-versus-adaptive question. We therefore include it as a secondary baseline, with its gain tuned under the same budget, and apply our scheduling rule to that gain as well. If the scheduled filter gains as much as the scheduled Riccati weight, the contribution is the scheduling rule rather than the Riccati embedding, and we would say so.
Decisive Checks
The primary test is the safety–agility Pareto comparison against the tuned static pad. It is run on the planar quadrotor in cluttered navigation (the BaS-SDRE setting) and on the Crazyflie flying the 3D Eight-Shape through wall and gate constraints. It covers perturbed simulators and at least 30 hardware flights per method. Metrics are collision and violation rate, minimum margin, completion time, mean tracking error, peak command, and command smoothness.
The mechanism check correlates $\lambda_j(t)$ with ground-truth model error in simulation. A weak correlation would mean the gains come from the PID term alone, which the ablation isolates.
The oscillation check measures limit cycles in $\lambda$ with and without the derivative term. The literature shows undamped multipliers cycle between reckless and frozen behavior, and the check tests whether that carries over to physical time.
Finally, we report Riccati solve latency on the flight computer. If it exceeds the control period at the required rate, the method is not deployable as proposed, whatever its other results.
References
- Learning Control-Oriented Dynamical Structure from Data
- Barrier-Riccati Synthesis for Nonlinear Safe Control with Expanded Region of Attraction
- Provably Safe Sim-to-Real Transfer
- Safety Gymnasium: A Unified Safe Reinforcement Learning Benchmark
- Linearized alternating direction method with parallel splitting and adaptive penalty for separable convex programs in machine learning
- Self-Supervised Primal-Dual Learning for Constrained Optimization
- Constrained Differential Dynamic Programming: A primal-dual augmented Lagrangian approach
- What Matters in Learning a Zero-Shot Sim-to-Real RL Policy for Quadrotor Control? A Comprehensive Study
- QuaDUE-CCM: Interpretable Distributional Reinforcement Learning using Uncertain Contraction Metrics for Precise Quadrotor Trajectory Tracking
- OmniDrones: An Efficient and Flexible Platform for Reinforcement Learning in Drone Control
- The Power of Input: Benchmarking Zero-Shot Sim-to-Real Transfer of Reinforcement Learning Control Policies for Quadrotor Control