Catching Surrogate Exploitation in Quantum Control with Physical Self-Consistency

Designing control pulses for qubits that sit in a warm environment is expensive. Each candidate pulse has to be checked by numerically solving the equations for an open quantum system, meaning one that leaks coherence into a finite-temperature bath. Erratic trial pulses make those equations stiff and the solver slow. Graphics researchers faced the same kind of bottleneck with slow, non-differentiable renderers. Their answer was to train a neural network to imitate the expensive simulator and then optimize through the imitation. Reward-hacking research predicts what happens when you try this in quantum control: an optimizer pushed hard against an imperfect learned model finds inputs where the model is wrong in its own favor. Here that means a pulse that looks like near-perfect cooling or gate fidelity inside the network and fails on the real dynamics.

This note proposes a way to detect and suppress that exploitation that is not available for language-model reward hacking. The true dynamics of a thermally coupled quantum system obey exact identities that can be checked without calling the solver. Running a pulse in two halves must give the same result as running it whole. The output must be a valid quantum state. For Gaussian systems such as an optomechanical resonator, the final phonon count must be exactly linear in the bath's thermal occupation, for any pulse. A learned simulator that is being exploited is being driven into regions it never saw in training, and there its errors should break these identities. We propose measuring these violations as a free, solver-free signal that "the model is being fooled here." We then use that signal to penalize the optimizer and to decide when to spend a real solver call.

Core Claim

Our claim is that, for neural surrogates of thermal open-system dynamics, violations of exact physical identities track true surrogate error better than standard model-uncertainty signals, specifically along the trajectories an optimizer actually follows. The three identities are composition of the evolution over time, physical validity of the output, and exact affine dependence on bath temperature where it holds. If this is true, a surrogate-based pulse optimizer guarded by these violations should reach the true-solver fidelity of direct solver-based control with far fewer solver calls. It should also avoid the hard smoothness limits that current methods use to stay safe, limits that rule out fast pulses.

In RLHF there is no cheap check on the true objective. Detecting hacking requires another learned judge, and Style Outweighs Substance shows those judges are themselves gameable. Physics supplies checks that are exact rather than learned, and that difference is what we exploit.

Where This Sits

The motivation comes from three findings. First, Reinforcement Learning for Quantum Control under Physical Constraints trains constrained Proximal Policy Optimization (PPO) directly against open-system solvers in JAX. It identifies solver stiffness from erratic pulses as the central cost. It handles that cost with a solver step cap, Gaussian filtering of pulses and bandwidth penalties, and it acknowledges that these constraints may exclude fast non-adiabatic pathways. It never replaces the solver with a learned model, so it never meets surrogate exploitation. Second, Optimal control of linear Gaussian quantum systems via quantum learning control avoids learned models entirely. It differentiates exact moment equations, which works only because Gaussian systems are fully described by their means and covariances. Third, reward-hacking work documents the failure we expect. Loose lips sink ships shows PPO exploiting a learned reward model's length bias, and the problem shifts elsewhere once one shortcut is removed.

The method comes from differentiable proxy rendering. Node Graph Optimization Using Differentiable Proxies replaces non-differentiable procedural nodes with StyleGAN2 imitators. It then re-inserts the true generator for a final refinement stage, and it reports divergence when parameters leave the proxy's training range. Neural Rendering for Game Character Auto-Creation found that unconstrained descent through its renderer imitator produced grotesque faces until parameters were confined to a PCA subspace of real faces.

Both of these, and the Ernst et al. filter, prevent exploitation by shrinking the search space. That is the tension the file's Gap 2 identifies. Thermal physics has sharp thresholds: René et al. report a thousand-fold collapse in hole-spin coherence between roughly 8 K and 20 K, and Storcz et al. show that optimal gate settings shift qualitatively with temperature. Smooth low-dimensional priors risk excluding the very pulses that work in such regimes. Our approach does not restrict where the optimizer may go. It restricts how much the optimizer trusts the surrogate where it goes. To our knowledge from the file, no work trains a neural surrogate of thermal open-system dynamics for control, and none studies exploitation of such a surrogate.

Research Program

We train a surrogate $F_\phi$, a neural network with weights $\phi$. Its inputs are a control pulse segment $u_{[a,b]}$ (the drive amplitude and phase between times $a$ and $b$) and a bath thermal occupation $\bar n$. Its output is the propagator of that segment, a matrix acting on the vectorized density matrix. Predicting the propagator rather than the final state makes the evolution linear in the initial state by construction. It also lets us state the composition identity directly. For time-local thermal dynamics, the true propagator over $[0,t_2]$ equals the propagator over $[t_1,t_2]$ multiplied by the one over $[0,t_1]$. The surrogate's violation is

$$r_{\text{comp}}(u) = \big\| F_\phi(u_{[t_1,t_2]},\bar n)\,F_\phi(u_{[0,t_1]},\bar n) - F_\phi(u_{[0,t_2]},\bar n) \big\|,$$

averaged over randomly drawn split points $t_1$.

We add two further residuals. The validity residual $r_{\text{valid}}$ measures how far the predicted map is from preserving trace and positivity on a fixed set of probe states, and how far a long drive-off segment lands from the thermal Gibbs state at $\bar n$. For Gaussian systems we add $r_{\text{temp}}$, the deviation of the predicted final phonon number from linearity across a ladder of bath occupations. This identity is exact there because the covariance equation is linear and its noise source is affine in $\bar n$.

Pulse optimization minimizes the control loss through the surrogate plus a penalty on these residuals:

$$J(u) = \mathcal{L}\big(F_\phi(u)\big) + \lambda \big(r_{\text{comp}} + r_{\text{valid}} + r_{\text{temp}}\big),$$

where $\mathcal{L}$ is infidelity or phonon number and $\lambda$ sets the penalty strength. The residuals serve a second purpose. When their total exceeds a threshold calibrated on held-out solver data, we call the true solver on the current pulse and add the result to the surrogate's training set. This is active learning triggered by self-inconsistency rather than by a fixed schedule.

The analysis has three parts. First, we measure whether naive optimization through the surrogate actually hacks it. We define the hacking gap as surrogate-predicted fidelity minus true-solver fidelity along optimization, and we characterize the exploiting pulses: their spectral content, and whether they cluster near the edges of the training temperature range. Second, we measure how well each residual predicts true surrogate error on optimizer trajectories, compared against model-uncertainty signals. Third, we run the full guarded optimizer and count solver calls.

Baselines

The naive reference point is gradient descent through the same surrogate with no residual penalty and no solver re-queries, followed by evaluation on the true solver. Its hacking gap defines the problem. If that gap is small, the premise fails, and we report that as the finding.

The strongest existing method is constrained PPO on the native solver from Reinforcement Learning for Quantum Control under Physical Constraints. It reports fidelities above 0.999 across its three systems and a transmon reset fidelity of 0.9997, against 0.983 for existing experimental pulses. We rerun it on the same simulated Λ system, Rydberg gate and transmon reset, using its JAX setup. We also run the same objective by direct gradient descent through the solver, so that "solver versus surrogate" is not confounded with "RL versus gradients." For the Gaussian setting, the reference is exact moment-equation control from Optimal control of linear Gaussian quantum systems. It reports cooling from about $10^3$ thermal phonons to $\bar N_b \approx 0.05$, and logarithmic negativity of 0.6 to 0.9 from a 100-phonon bath, against a steady-state value below 0.3.

Beating these baselines does not require higher fidelity. It means matching their true-solver fidelity within error bars while using at least an order of magnitude fewer solver calls, counting the calls used to build the surrogate's training set. A second win condition is reaching higher fidelity than their filtered pulses once bandwidth limits are relaxed.

The strongest uncertainty-aware competitor is a deep ensemble of five surrogates whose disagreement replaces our residuals in both the penalty and the re-query trigger. We give it the same total training compute as our single surrogate, so each member gets a fifth. We also run a refinement baseline modeled on the third stage of Node Graph Optimization Using Differentiable Proxies: naive surrogate optimization followed by fine-tuning on the true solver, given our total solver budget. Finally, a smoothness baseline applies the Ernst et al. Gaussian filter and bandwidth penalty to naive surrogate optimization.

Ablations isolate the key ingredient. We try each residual alone. We include a "random residual" control: noise with matched magnitude and timing that triggers re-queries equally often, which tests whether gains come from the information in the residuals or simply from extra solver calls. We also try a surrogate that predicts final states instead of propagators, so that linearity in the initial state becomes a further check rather than a built-in property. Every method gets identical solver-generated training data, the same hyperparameter search budget, and the same per-system solver-call cap, all reported.

No reported result was found on Lacuna for surrogate-based open-system quantum control, so the surrogate-side numbers will be new.

Why not just…?

Why not just use the true solver, as Ernst et al. do? For the small systems in that benchmark this is a genuine threat, because the solver is affordable there. The proposal's value then rests on two things. One is the solver-call savings. The other is whether, once pulses are allowed to be unsmoothed, guarded surrogate optimization finds fast pulses that the step cap would discard; Ernst et al. themselves flag this cost. We report both, and we state plainly if the call savings are small at this scale.

Why not just use ensemble disagreement, the standard uncertainty signal? This is the most serious alternative. Our reason to expect it to fall short is that ensemble members share an architecture and training data. Under optimization pressure they may extrapolate similarly into high-frequency pulse regions and agree on the same hack. Exact identities do not depend on the members disagreeing. This is an empirical claim, and the residual-versus-disagreement comparison on optimizer trajectories settles it. If disagreement predicts true error as well, we will recommend the combination or the simpler method, not our own.

Why not just constrain the pulse space, with filtering or a learned low-dimensional prior? Neural Rendering for Game Character Auto-Creation shows that subspace constraints do stop exploitation. The cost is expressiveness. In thermal regimes near sharp coherence thresholds, the useful pulses may lie outside a smooth prior. The smoothness baseline, evaluated in temperature sweeps that cross such thresholds, measures what that constraint gives up.

Why not just refine on the true solver at the end? If the surrogate optimum lands in a hacked region far from any good true optimum, local refinement inherits a bad starting point. This is the same local-minimum issue Node Graph Optimization Using Differentiable Proxies reports. The refinement baseline at equal solver budget tests this directly.

Decisive Checks

We will run the experiments in order of how much each could falsify the idea. First, measure the hacking gap of naive surrogate optimization on all three Ernst et al. systems and on Gaussian cooling. Without a substantial gap there is nothing to fix. Second, compute the rank correlation and detection performance of each residual against true surrogate error, restricted to points visited by the optimizer, compared with ensemble disagreement. This is where the core claim lives or dies. Third, plot true-solver fidelity against cumulative solver calls for every method. Fourth, sweep bath temperature across a threshold-like regime and test whether residual guarding keeps its advantage where the dynamics change sharply. This is the regime where smooth priors should fail and identity checks should not.