Value iteration finds the best long-run controller by repeatedly applying one update, the Bellman operator, until the value estimates stop changing. It is guaranteed to converge because a discount factor $\gamma<1$ shrinks the gap between any two value estimates at each step. Many quantum control problems are about holding a qubit or atomic ensemble in a target state forever, and there discounting is the wrong objective: it down-weights exactly the late-time decay one wants to prevent. Without discounting, the update stops shrinking errors. Iterates drift, oscillate, or converge slowly. Our claim is that in open quantum systems this lost shrinkage is often supplied by the physics. Decoherence, the leaking of quantum information into the environment, makes any two initial states of the system become harder to tell apart over time. That forgetting can play the role a discount factor normally plays.
The proposal makes this precise and tests it. We define a contraction modulus $\eta$ measured directly from the controlled noise channels. We prove that it bounds the convergence of undiscounted value iteration on density matrices, the matrices that describe mixed quantum states. We then build a solver that uses $\eta$ as an effective discount when it is below one. When it approaches one, the solver falls back on anchored iteration, a method designed for updates that do not shrink errors. The theory makes a sharp, falsifiable prediction. Energy relaxation (the $T_1$ process) should speed up planning, while pure dephasing (the $T_\phi$ process, which destroys phase information but leaves populations alone) should not. Near decoherence-free regimes, plain undiscounted iteration should stall, which is where anchoring becomes necessary.
Core Claim
Let $\Phi_u$ be the quantum channel, a linear map on density matrices, that evolves the state over one control interval $\Delta t$ under control $u$ and the Lindblad master equation (the standard model for an open system: unitary evolution plus jump operators that encode decoherence). Define
$$\eta = \sup_{u}\ \sup_{\rho\neq\sigma} \frac{\|\Phi_u(\rho)-\Phi_u(\sigma)\|_1}{\|\rho-\sigma\|_1},$$
the worst-case factor by which one control step shrinks the trace distance $\|\cdot\|_1$ between two states. A unitary step has $\eta=1$. An amplitude-damping step has $\eta<1$. A pure-dephasing step also has $\eta=1$, because it leaves every diagonal (classical) mixture unchanged and so preserves their distinguishability.
The undiscounted Bellman operator for steady-state reward $r(\rho,u)$ is $(\mathcal{T}V)(\rho)=\max_u[r(\rho,u)+V(\Phi_u\rho)]$. It is not a contraction in the sup norm, because adding a constant to $V$ leaves it unchanged. That is the shift invariance analyzed in Finite Sample Analysis of Average-Reward TD Learning and $Q$-Learning. For a fixed policy $\pi$ with channel $\Phi_\pi$, a short argument gives
$$\mathrm{Lip}(\mathcal{T}_\pi V-\mathcal{T}_\pi W)\le \eta\,\mathrm{Lip}(V-W),$$
where $\mathrm{Lip}$ is the Lipschitz seminorm with respect to trace distance. This seminorm ignores constant shifts. So policy evaluation is an honest $\eta$-contraction once the gauge freedom is removed, and the decoherence rate sets the rate.
The open and central question is control. There the maximum over controls breaks this one-line argument, just as the optimality operator breaks contraction in distributional RL (A Distributional Perspective on Reinforcement Learning, which shows evaluation contracts while control can oscillate). We conjecture a geometric rate for control, governed by a multi-step version of $\eta$, under a uniqueness-of-steady-state condition. We aim to prove it at least in the continuously monitored case, where measurement randomness should supply a Dobrushin-type mixing argument (bounding how much two different starting states still differ in distribution after one random step).
Positioning Against Prior Work
Two papers are closest. Accelerating Value Iteration with Anchoring introduces Anchored Value Iteration (Anc-VI), $U^k=\beta_kU^0+(1-\beta_k)\mathcal{T}U^{k-1}$. It proves an $O(1/k)$ Bellman-error rate even at $\gamma=1$, along with a $\gamma$-dependent schedule for $\gamma<1$. That analysis is worst-case over abstract MDPs, has no empirical study, and says nothing about where an effective contraction might come from. We reuse its method, but we feed the physically derived $\eta$ into its $\gamma$-dependent schedule rather than treating every problem as $\gamma=1$.
Stabilizing Quantum States by Constructive Design of Open Quantum Dynamics proves that global asymptotic stability of a target state is equivalent to a unique steady state of the master equation. It gives a constructive way to design the jump operators. It does not optimize convergence speed and never poses stabilization as dynamic programming. Its uniqueness condition is the hypothesis our control conjecture needs, so it motivates the theory, and its constructed dynamics serve as our naive physics baseline.
The average-reward literature constrains the design. Bridging the Gap Between Average and Discounted TD Learning restores unique fixed points for undiscounted evaluation through gradient splitting, but only for evaluation and linear features. Relative Value Learning removes the gauge with pairwise value differences, but still relies on $\gamma<1$ for contraction. Our Lipschitz-seminorm view is a third way to kill the gauge freedom, and it ties the rate to a measurable physical quantity.
On the quantum side, prior learning-based control is purely heuristic, with no Bellman analysis. Scaleable Quantum Control via Physics Constrained Reinforcement Learning uses PPO with solver-step caps and low-pass reward shaping on episodic discounted tasks. QMaxCal uses path-space penalties from Girsanov's theorem, which compares the probability of measurement records under different drifts. Efficient Optimal Control of Open Quantum Systems is open-loop gradient descent. None asks how the noise model shapes the geometry of the planning problem, which is the question here.
Research Program
The theoretical work has three parts. The first is the evaluation result above, together with an explicit computation of $\eta$ for qubit channels combining amplitude damping, dephasing, and coherent drive. This yields the predicted formula: $\eta$ falls with $T_1$ as roughly $e^{-\Delta t/(2T_1)}$ (the trace-distance shrinkage of amplitude damping) and does not depend on $T_\phi$. The second is the control conjecture, attacked in two settings. In the unmonitored case the state evolves deterministically. In the homodyne-monitored case the state follows the stochastic Schrödinger equation and the controller sees the conditioned state. The third addresses the gap that Accelerating Value Iteration with Anchoring leaves open between Bellman error and distance to the solution. We will bound the steady-state infidelity of the greedy policy in terms of the Lipschitz-seminorm Bellman residual, so that the solver's stopping criterion means something physical.
The algorithmic work produces one solver, which we call gap-aware anchored value iteration. It first upper-bounds $\eta$. For small systems it maximizes the contraction ratio over sampled state pairs and controls. For larger ones it uses a multi-step coefficient when the one-step value is 1. It then runs Anc-VI with the schedule matched to $\eta$. Value functions live on a grid over the Bloch ball for one qubit, and on a neural approximator trained by fitted iteration for the larger benchmarks. The ablations swap the $\eta$-matched schedule for the $\gamma=1$ schedule and for no anchoring.
Baselines
The naive reference point is the fixed engineered dissipation from Stabilizing Quantum States by Constructive Design of Open Quantum Dynamics where it is realizable with the available controls, and otherwise constant resonant drive. Any learned controller must beat this on time-averaged steady-state fidelity to be worth anything.
The planning baselines share one simulator, one control discretization, and one compute budget, measured as Bellman applications and GPU wall-clock time. They are:
- Discounted value iteration with $\gamma\in\{0.99,0.999\}$.
- Classical relative value iteration, which subtracts the value at a reference state each step.
- Anc-VI with its worst-case $\gamma=1$ schedule.
The last is the ablation that isolates our key ingredient: using the physical $\eta$ in place of the worst case. Beating it means fewer iterations to a given residual whenever $\eta<1$, with no loss when $\eta\to1$.
The strongest existing method is constrained PPO from Scaleable Quantum Control via Physics Constrained Reinforcement Learning. We run it in its own JAX transmon-reset and $\Lambda$-system environments, with its reported hyperparameters and an equal wall-clock budget. That paper reports a transmon reset fidelity of 0.9997 and infidelities below $10^{-3}$ on all three systems. It also reports 39 seconds to reach 0.99 fidelity, against 4,517 seconds for TD3. Those are episodic final-state fidelities. Our metric is time-averaged fidelity over a long hold window, so we re-evaluate the PPO policies under that metric rather than quoting their numbers as comparable.
Beating PPO means matching its fidelity within seed-to-seed error while also delivering a certified residual bound, or exceeding it during the hold phase, where an episodic discounted objective has no incentive to fight late decay. For the 6-qubit IBM Kingston chain in QMaxCal, which reports about 16% infidelity reduction, dynamic programming over density matrices is out of reach. We list it only as a scope boundary.
Why not just…?
Why not just use discounted value iteration with $\gamma$ close to 1? Its geometric rate degrades as $\gamma\to1$ (Accelerating Value Iteration with Anchoring). High-discount TD also failed to recover the average reward on battery dispatch in Bridging the Gap Between Average and Discounted TD Learning. The honest threat is different: if our thesis is right, strong $T_1$ already makes the problem easy, and a moderate $\gamma$ might give near-optimal policies cheaply. The qubit sweep settles this by reporting the steady-state fidelity gap between discounted-optimal and average-optimal policies as a function of $T_1$ and $\gamma$. If that gap is negligible everywhere, the undiscounted machinery matters only near decoherence-free regimes, and we will say so.
Why not just run classical relative value iteration? This is the most serious alternative. Our own theory predicts it converges geometrically whenever $\eta<1$ is bounded away from 1. The proposal's value then lies in predicting when it works, from $T_1$ and $T_\phi$ alone, and in handling the regime where it fails. The test is the near-dark-state $\Lambda$ system and weakly damped drives. There we expect relative VI to stall or oscillate while Anc-VI keeps its $O(1/k)$ rate. If relative VI never stalls even there, anchoring is unnecessary and the contribution reduces to the convergence-rate theory.
Why not just use constrained PPO, which already reaches fidelities above 0.999? It may well match us on fidelity, and the comparison above is set up to show that honestly. What it cannot give is a convergence certificate or a prediction of how hard a new noise model will be. Its discounted episodic objective also has no term for holding a state indefinitely. The hold-window evaluation tests whether that omission costs fidelity.
Why not just engineer the dissipation analytically? Constructive reservoir design gives stability but does not optimize convergence speed (Stabilizing Quantum States by Constructive Design of Open Quantum Dynamics). Passive pumping slows as it approaches fundamental limits, leaving room for unmodeled decay (Dissipative Preparation of Spin Squeezed Atomic Ensembles in a Steady State). It is our floor baseline. Beating it on time-averaged fidelity under the same hardware-limited controls is a minimum requirement.
Proposed Experiments
The decisive experiment is a single-qubit study on a Bloch-ball grid, with discretized drive amplitude and phase, run both unmonitored and with homodyne monitoring. We sweep $T_1$ and $T_\phi$ independently across several orders of magnitude, recording $\eta$, the number of iterations needed to reach a fixed Lipschitz-seminorm residual, and the steady-state fidelity of the greedy policy. The thesis is confirmed only if three things hold. Iteration counts must track $1/(1-\eta)$ as $T_1$ varies. They must stay flat as $T_\phi$ varies at fixed $T_1$. And the residual bound must correctly upper-bound the measured infidelity. A dependence on $T_\phi$ comparable to that on $T_1$ would falsify the claim that trace-distance contraction is the operative mechanism.
The second experiment moves to the transmon-reset and $\Lambda$ environments of Scaleable Quantum Control via Physics Constrained Reinforcement Learning, using fitted gap-aware anchored iteration against all the baselines above, at equal budget and over multiple seeds. The third experiment deliberately drives the $\Lambda$ system toward its dark state, where $\eta\to1$. There the predicted ordering is: Anc-VI with the $\eta$-matched schedule at least as good as the worst-case schedule, both converging, and relative and discounted iteration stalling. A failure of that ordering would show that anchoring adds nothing beyond what decoherence already provides.
References
- Finite Sample Analysis of Average-Reward TD Learning and $Q$-Learning
- A Distributional Perspective on Reinforcement Learning
- Accelerating Value Iteration with Anchoring
- Stabilizing Quantum States by Constructive Design of Open Quantum Dynamics
- Bridging the Gap Between Average and Discounted TD Learning
- Relative Value Learning
- Scaleable Quantum Control via Physics Constrained Reinforcement Learning
- QMaxCal: Path-Space Regularization for Open Quantum Control via Girsanov's Theorem
- Efficient Optimal Control of Open Quantum Systems
- Dissipative Preparation of Spin Squeezed Atomic Ensembles in a Steady State