Solver-Aware Audits for Nonlinear Control Benchmarks

When a controller for a nonlinear robot is benchmarked, the reported number, such as tracking error or constraint violations, is usually treated as a property of the algorithm. For controllers built on model predictive control (MPC), where the controller solves an optimization problem at every time step to choose its next action, the number also depends on how that optimization was actually solved on that machine: which solver version ran, whether it hit its iteration cap, what tolerance it settled for, and how much wall-clock time it had. On nonlinear systems these optimizations are non-convex, so the solver quietly returns "best effort" answers far more often than on linear ones. We propose to measure how much of the reported progress on nonlinear safe control is really a property of the execution environment and the solver's hidden behavior. To do this we would build a harness that runs each benchmark as an isolated, version-pinned service stepped in closed loop, and that records, at every control step, whether the solver actually did what the stability and safety arguments assume.

The core bet has two parts. First, controller rankings on standard nonlinear safe-control tasks will change under realistic, one-factor-at-a-time changes to the execution environment. Second, a cheap per-step record of solver integrity (convergence status, iterations used, constraint residual, time spent) predicts transient safety violations that outcome-only metrics miss. If rankings turn out to be stable and solver flags carry no predictive signal, that is also worth knowing, because it would validate the existing evaluation practice.

Core Claim

"Generalization to nonlinear systems" in control mostly means carrying guarantees such as stability, recursive feasibility, and constraint satisfaction from linear plants to nonlinear ones. The guarantees in this literature assume something benchmarks never check: that the online optimization was solved to the stated accuracy. The linear robust adaptive MPC of Linear robust adaptive model predictive control: Computational complexity and conservatism is real-time precisely because its tube problem is small and convex (30 optimization variables versus 18,173 for homothetic tubes), and its proofs are restricted to linear plants. Once the plant is nonlinear, as with the Cart-Pole and 2D quadrotor in safe-control-gym, the MPC problem becomes a non-convex program. An interior-point solver such as IPOPT, the default behind CasADi-based MPC, can then stop early under an iteration cap or deadline and still hand back a usable-looking action. Audit-Grade Harness for Agent-driven Scientific Computation Workflows named this failure pattern "silent degradation" for scientific pipelines that swap in a coarser mesh or cheaper solver while still reporting success. Our claim is that the same pattern exists in nonlinear control benchmarks, that it can be measured, and that measuring it changes conclusions.

Relation to Prior Work

Four pieces of prior work motivate the gap, and they fall into two pairs. Reproducibility of Benchmarked Deep Reinforcement Learning Tasks for Continuous Control showed that on MuJoCo Half-Cheetah, TRPO's average return ranged from roughly 1,000 to over 5,000 across architecture and batch-size choices. That study concerned learning hyperparameters and seeds, not the numerical execution of an online optimizer, and it proposed no instrument for isolating causes. safe-control-gym (paper, journal version; Safe-Control-Gym: A Unified Benchmark Suite for Safe Learning-Based Control and Reinforcement Learning in Robotics) supplies what we build on: nonlinear dynamics in PyBullet, symbolic CasADi models and constraints, a disturbance-injection interface, and baselines including LQR, linear MPC, GP-MPC, PPO, SAC, a safety layer, and control barrier functions. It runs from a local Python install and scores outcomes and violation rates, with no record of solver behavior.

The infrastructure side supplies the method. Bencher puts more than 80 benchmarks, including MuJoCo Ant, Hopper, and Walker, into Docker/Singularity containers behind a gRPC interface. It supports only one-shot, unconstrained black-box evaluation, not step-by-step closed loops with constraints and disturbances. Agent-Diff scores agents by the difference between initial and final container state. We borrow the idea of a declarative contract but must make it trajectory-aware, since a before/after diff cannot see a quadrotor briefly entering a keep-out zone. SciAgent contributes version-pinned service registries and append-only logs, and it also constrains our design: its verifier is a language model whose confidence is not calibrated. Every check we need is numerical, so our verifier is deterministic code. Bootstrapped Model Predictive Control ships a Docker setup purely as an installation convenience, which illustrates the common case where containers support installation but are not used as an experimental control.

Research Program

We would build a closed-loop control service. Each benchmark environment, meaning the safe-control-gym tasks and a secondary DMControl subset via the BMPC/TD-MPC2 container (wertyuilife2/bmpc), runs in its own pinned container and exposes reset, step, and disturbance-schedule calls over RPC. The controller runs in a separate container. This separation is the point of using containers: it lets us change exactly one factor at a time while holding everything else bit-identical. The factors are the physics engine version, the CasADi/IPOPT version, the linear-algebra backend and thread count, the solver iteration cap and tolerance, and the per-step wall-clock budget enforced by the harness.

Every step appends one record to a log. The record contains the full state, the action, the injected disturbance, and a solver-integrity entry: termination status, iterations used, final primal and dual residual, and solve time.

A trajectory contract, written declaratively per task, turns the log into verdicts. The first verdict is outcome success. The second is the maximum constraint violation over the whole trajectory, not just the final state. The third is solver-integrity compliance, defined as the fraction of steps in which the solver terminated at its stated tolerance within the budget. The verifier replays each logged trajectory in a fresh container from the logged initial state and disturbance sequence and flags any divergence. This replay check is how we detect cross-host non-determinism.

The analysis asks three questions. First, how stable are controller rankings across the factor grid, measured by Kendall's rank correlation between the default configuration and each single-factor variant, for each metric? Second, do solver-integrity flags predict constraint violations in the following $k$ steps, measured by the area under the ROC curve when the solver-status features alone are used as the predictor? Third, what does the RPC boundary cost at control rates of 100 Hz to 1 kHz? The research file notes that the timing side of this question is uncharacterized for containerized control. The neural controller of Imitation Learning with Stability and Safety Guarantees, at 0.01 s per step versus 0.38 s for an explicit MPC expert, sets the latency scale that a service boundary must not dominate.

Baselines

The naive reference point is stock safe-control-gym run from a local install with default solver settings and scored only by its native outcome metrics. This is how results in the area are currently reported. Every other condition is compared against it on the same tasks, disturbance schedules, and seeds.

The strongest existing evaluation protocol is safe-control-gym's own: tracking error and constraint-violation rate for GP-MPC, linear MPC, nonlinear MPC, PPO, PPO with a safety layer, and CBF-filtered controllers. We will use the authors' hyperparameters with no extra tuning for any method, and give every controller the same data budget and the same per-step compute budget. The reported results we can anchor to are qualitative or relative. GP-MPC reaches optimal performance with about 1/10 of the data PPO or SAC need, and corrects up to 50% model error within seconds (safe-control-gym). GP-MPC and learned CBF methods reach low violation rates with fewer examples than PPO or PPO with a safety layer (Safe-Control-Gym). We found no reported tracking-error or violation-rate figures on Lacuna for these tasks, so our first deliverable is to reproduce them under the default configuration. The audit "beats" this protocol if it changes a conclusion the protocol would draw. That means either a ranking flip with a rank correlation well below 1 under a single realistic factor change, or a significant fraction of trajectories that pass the outcome metric but fail the whole-trajectory or solver-integrity contract.

For the secondary high-dimensional check, the baseline is BMPC's reported advantage: it reaches the solve threshold on the 7 hardest tasks in 90k steps versus 360k for TD-MPC2 (Bootstrapped Model Predictive Control). We will test whether this gap survives when the planner's sampling budget is tied to a fixed wall-clock deadline instead of a fixed sample count.

Two ablations isolate the key ingredient. The first removes the solver-integrity channel and keeps only the whole-trajectory constraint contract. This tells us whether solver status adds predictive value beyond simply watching the state. The second replaces container isolation with a lock-file Python environment on each host. This tells us how much of the cross-host divergence containers actually remove.

Why not just…?

Why not just pin dependencies with a Conda or pip lock file? This is a real threat to the infrastructure half of the proposal, and we treat it as an experiment rather than a settled point. Bencher and the Shifter deployments in BOSS-LDG argue that lock files miss system libraries and compiler layers. For a PyBullet/CasADi stack, lock files might still be enough in practice. The second ablation measures bitwise trajectory divergence across heterogeneous hosts under both regimes. If lock files match containers, we say so and keep containers only because they make single-factor swaps cheap. The solver-audit findings do not depend on containers.

Why not just run more seeds and report confidence intervals? Seed variance, which Islam et al. documented, is a different axis from solver-execution variance. A controller can be perfectly deterministic across seeds and still change behavior when IPOPT's iteration cap or the step deadline changes. Our design crosses seeds with execution factors, so the variance decomposition directly shows how much of the spread each accounts for. If seeds dominate, the proposal's main claim weakens.

Why not just log the solver return code inside existing safe-control-gym code? Mechanically, much of the audit is exactly that, and we consider its cheapness a feature. What a code patch alone does not give is the controlled factor grid needed to show that solver behavior drives ranking changes, the replay verifier that catches cross-host divergence, and a shared contract that makes results comparable across groups. If the prediction experiment shows that solver flags carry no signal, the simple patch is also unnecessary, and we would report that.

Why not just give the solver unlimited iterations and remove the problem? Doing so evaluates a different controller from the one a real-time system would deploy. The guarantees these methods advertise matter precisely under deadlines. We include the unlimited-budget condition as the upper reference. The gap between it and deadline-bound execution is one of the quantities the study reports.

Proposed Experiments and Decisive Checks

The main experiment runs every safe-control-gym baseline on Cart-Pole stabilization and on 2D-quadrotor tracking of the circle, figure-eight (lemniscate), and square references. Each run uses the benchmark's parametric perturbations, white noise, step gusts, and impulse disturbances, crossed with the single-factor execution grid and 20 seeds per cell. The secondary experiment repeats a reduced grid on the BMPC DMControl tasks.

The claim fails if all of the following hold: rank correlations stay near 1 across realistic factor changes; fewer than a small fraction of outcome-passing trajectories fail the whole-trajectory contract; and solver-status features predict upcoming violations no better than chance once current state is controlled for. The claim succeeds if some realistic factor, such as an IPOPT minor-version change or a 2x tighter step deadline, reorders nonlinear MPC relative to GP-MPC or CBF-filtered RL, and solver-integrity flags anticipate the violations responsible for the reordering. The latency measurement settles a separate practical question: whether a containerized step boundary stays well under the 10 ms scale of the neural controllers above at 100 Hz. If it does not, the harness supports offline replay-based audits only, not live real-time evaluation.