State-Agnostic Safety Certificates for Exploring Quantum Control Policies

Reinforcement learning can now find laser and microwave pulses that drive small quantum systems with fidelity above 0.999. The trial and error that produces those pulses is harmless in simulation, but it becomes a real problem once learning runs in closed loop on hardware. An exploratory pulse can pump population into energy levels outside the computational subspace, a failure called leakage, and that corrupts every later operation. The obvious fix, a "shield" that checks each proposed action before it runs, meets a quantum-specific obstacle. The controller never knows the current quantum state exactly, because reading the state destroys it. A shield that simulates forward from a best-guess state certifies only that guess.

This note proposes a shield built on the quantum weakest precondition. The weakest precondition is a formal-verification tool that runs a safety requirement backward through a program instead of pushing states forward. For a quantum operation, running a measurable quantity backward yields a single operator. Its expectation on any input state predicts the post-action value of the safety quantity. One backward pass therefore certifies an action for every state the controller might be in, and checking the certificate reduces to an eigenvalue computation. We wrap this certificate in a projection step that nudges unsafe exploratory pulses to the nearest certified ones. Parameter uncertainty and state uncertainty are both handled by explicit margins. We test the shield on open-system quantum control benchmarks.

Core Claim

Quantum dynamics are linear in the density matrix, the matrix that describes a possibly noisy quantum state. As a result, the weakest precondition of a linear safety constraint is again a single linear constraint, and it holds uniformly over a whole set of states. Let $\Phi_a$ be the noisy evolution induced by pulse segment $a$, $M$ a safety observable such as the projector onto leakage levels, and $c$ the allowed leakage budget. The requirement after the pulse becomes a requirement before it:

$$\mathrm{tr}(M\,\Phi_a(\rho)) = \mathrm{tr}(W_a\,\rho), \qquad W_a = \Phi_a^{\dagger}(M),$$

where $\Phi_a^\dagger$ is the adjoint map acting on observables (the Heisenberg picture). Two consequences follow. If the controller only knows its state lies within trace distance $\delta$ of an estimate $\hat\rho$, the worst case is bounded in closed form:

$$\sup_{\|\rho-\hat\rho\|_1\le\delta}\mathrm{tr}(W_a\rho)\;\le\;\mathrm{tr}(W_a\hat\rho)+\tfrac{\delta}{2}\big(\lambda_{\max}(W_a)-\lambda_{\min}(W_a)\big).$$

This follows because $\rho-\hat\rho$ is traceless, so $W_a$ may be shifted by any multiple of the identity before applying Hölder's inequality. If nothing is known about the state, the condition is simply $\lambda_{\max}(W_a)\le c$.

The second consequence concerns uncertain physics. When the Hamiltonian or decoherence parameters $\theta$ are known only to lie in an interval, the adjoint of a physical evolution cannot increase operator norm. A Grönwall-type argument then bounds $\|W_a(\theta)-W_a(\theta')\|$ by $\|M\|$ times the time-integrated difference of the generators. That bound gives a robust margin over the interval without enumerating it.

The claim to be tested is specific. A shield that searches for safe actions by linearization but accepts them only after this exact certificate check will do two things on closed-loop quantum control with uncertain states and parameters. It will cut leakage violations during training compared with constrained policy gradient and with forward-simulation shields. It will also leave final gate fidelity essentially unchanged.

Relation to Prior Work

The method comes from two sources. Quantum weakest preconditions formalized predicates as Hermitian operators with eigenvalues in $[0,1]$ and defined backward transformers through completely positive maps. It was used to verify programs such as Grover's search, never to steer a learner. Our $W_a$ is exactly their transformer applied to one pulse segment, and leakage projectors fit their bounded-predicate restriction. The known inability to represent unbounded observables therefore does not bite here. Guiding Safe Exploration with Weakest Preconditions (SPICE) runs classical weakest preconditions through a learned, locally linear model and projects actions with a quadratic program. It reports an 89% violation reduction versus CPO. Its documented weaknesses are violations caused by linearization error and poor safety at the start of training.

We differ from SPICE in three ways. We use operator predicates over density matrices rather than polyhedra over states. We handle unobserved states through the uniform-over-states property. We separate the approximate search from an exact acceptance test, so linearization can slow the search but cannot make an accepted action unsafe under the model. Neurosymbolic Reinforcement Learning with Formally Verified Exploration (REVEL) shows that verified shields can reach zero violations. It needs a worst-case dynamics model, and synthesis takes about 87% of training time. We borrow its fallback-action discipline but not its piecewise-linear invariants.

The application regime and its gap come from Reinforcement Learning for Quantum Control under Physical Constraints. That paper enforces physical feasibility with solver-step rejection, smoothing filters and reward penalties, and it gives no formal guarantee on leakage or state safety. We keep its constrained, filtered PPO as the base learner and add the shield on top.

Other papers set constraints on the design. Image Computation for Quantum Transition Systems and VeriQR report that exact quantum verification strains beyond roughly 20 qubits. We therefore restrict claims to the few-level systems in which control is actually learned, where $W_a$ is a small dense matrix. Breaking Safety Paradox with Feasible Dual Policy Iteration motivates a secondary benefit. As policies get safer, learned safety estimators receive fewer unsafe samples near the boundary. The certificate instead reports a margin, $c$ minus the bound, computed without sampling any unsafe state.

Research Program

We will build the shield in JAX on top of the RL4qcWpc simulators. At each control segment, the policy proposes pulse parameters. The shield integrates the adjoint Lindblad equation (the equation for noisy, dissipative quantum evolution, run in the observable picture) backward over the segment for the nominal parameters. It then evaluates the state-uncertainty bound above and adds the parameter-robustness margin. If the result exceeds $c$, the shield takes gradient steps on pulse parameters, obtained by differentiating through the adjoint solve, toward the nearest point that satisfies the certificate. The top eigenvector supplies a subgradient of $\lambda_{\max}$.

Every candidate is re-checked exactly before execution. If no certified pulse is found within a fixed iteration budget, a pre-verified idle or hold pulse runs instead.

For multi-step safety over a horizon of $H$ segments, we compose adjoints backward. The weakest precondition of "stay under budget at every step" is then the conjunction of $H$ linear operator constraints, each checked the same way.

The parameter interval is maintained online. We use a Bayesian estimate of detuning and decay rates, updated from the simulated measurement records the closed-loop setting already provides, and take a credible interval from it. The guarantee is explicitly conditional on that interval containing the truth. We will measure how often it does not.

On the analysis side, we will write down the robustness margin for each benchmark's generator. We will also report how conservative it is relative to brute-force worst case over a dense parameter grid.

Baselines

The naive reference is the constrained PPO of Reinforcement Learning for Quantum Control under Physical Constraints, run in closed loop with no shield and leakage entering only as a reward penalty. That paper reports fidelities exceeding 0.999 on all three systems, transmon reset fidelity of 0.9997 against 0.983 for the experimental pulse, and convergence up to 100x faster than TD3 and DDPG. Those are the fidelity targets a shielded learner must not fall meaningfully below. The paper reports no leakage-violation counts: no reported result found on Lacuna for training-time constraint violations on RL4qcWpc. We will measure these ourselves.

The strongest existing constrained-RL competitor is a Lagrangian variant of the same PPO, with PID-damped multipliers as in SafePO from Safety Gymnasium. There, PPO-Lag cut violations 98% relative to PPO at a 45% reward drop on velocity tasks, so it is a serious safety competitor with a known cost.

The strongest shield competitor is a forward-simulation shield. It rolls $\hat\rho$ forward under the nominal model and, in a stronger version, also under $K$ sampled states from the belief ball and $K$ sampled parameter values. We match $K$ so its wall-clock cost equals ours.

Two ablations isolate the key ingredients. A SPICE-style version accepts the linearized projection without the exact check; this isolates search/check separation. A point-state version certifies only $\hat\rho$ with nominal parameters; this isolates the uniform-over-states and robustness margins.

All methods share the base policy, filters, solver-step constraint, environment seeds, number of environment steps, and a hyperparameter search of equal size.

Beating the baselines means two things. First, at least an order of magnitude fewer training-time leakage violations than PPO-Lag and the forward shield under state and parameter uncertainty. Second, final fidelity within $10^{-3}$ of the unshielded learner.

Why not just…?

Why not just simulate forward from the estimated state? When the state is known exactly and the model is correct, forward simulation and our certificate agree, and ours offers no safety advantage. The difference appears only under uncertainty. Forward checks from samples can miss the worst-case state or parameter, while the eigenvalue bound cannot. This is a real threat, so the decisive experiment varies the belief radius $\delta$ and the parameter interval width. We predict parity at $\delta=0$ and a widening gap as uncertainty grows. If the sampled forward shield matches us at equal compute across that range, the proposal's central claim fails.

Why not just use a Lagrangian constraint? Lagrangian methods constrain expected cost at convergence rather than each exploratory step. REVEL reports thousands of violations for CPO and DDPG during training, and SPICE reduces CPO's violations by 89%. Our PPO-Lag baseline tests whether this pattern carries over to quantum control.

Why not just learn a neural safety critic? A learned critic needs observed leakage events to place its boundary. SPICE reports 73% fewer violations than the neural shield CSC-MBPO. The safety-paradox work further shows that such estimators degrade as the policy gets safer. We include a learned critic trained on the same data and report violations over time, especially late in training.

Proposed Experiments

We will use the three RL4qcWpc systems, each with a leakage observable: the excited intermediate level in the three-level Λ system, the intermediate Rydberg level in the two-qubit controlled-Z gate, and higher transmon levels in the reset task.

The environment's true parameters are drawn inside the shield's interval in the in-distribution split, and deliberately outside it in a mismatch split. We report cumulative leakage violations during training, final infidelity, wall-clock overhead of the shield, fallback frequency, and the certificate's conservativeness against grid worst case.

The decisive checks are three. The uncertainty sweep against the forward shield settles whether state-agnostic certificates matter. The no-exact-check ablation shows whether separating search from verification removes SPICE-style linearization violations. The mismatch split measures how gracefully guarantees degrade when the parameter interval is wrong. That condition is exactly where REVEL's proofs fail outright, and we expect margins to fail gradually, in proportion to the size of the mismatch.