Training-Only Quantum Critics for Decision Transformers

Quantum reinforcement learning has a placement problem. Most hybrid designs put a variational quantum circuit (a small programmable quantum circuit whose gate angles are trained like neural network weights) inside the policy, the part of the agent that picks actions. That forces a quantum computer, or a slow classical simulation of one, into every decision the agent makes. This proposal moves the circuit to a component that is used only during training and then thrown away. That component is the critic, the learned scorer that tells a Decision Transformer which of its training trajectories reflect skill rather than luck. If a circuit with a few dozen parameters can do that job, the deployed agent is a purely classical transformer. The quantum part never has to run when the agent acts, and offline training lets every circuit evaluation be batched.

The scientific payoff is a cleaner test of a question the field has not settled. Hybrid quantum agents keep matching classical ones with far fewer parameters. But they only learn at all once a classical layer is bolted onto the circuit, so it is unclear whether the circuit contributes anything that a matched classical module would not. We build the critic behind one swappable interface. Behind it sit the quantum circuit, parameter-matched classical surrogates, and circuits with specific ingredients removed. On the same offline benchmark, this setup answers two questions: whether the critic's benefit to a Decision Transformer survives being shrunk to quantum-circuit scale, and whether the quantum structure is doing the work.

Core Claim

A Decision Transformer learns to imitate trajectories conditioned on the return it should achieve. It fails when high returns in the data came from random luck, and it cannot stitch good pieces of different trajectories together. The Critic-Guided Decision Transformer (CGDT) fixes this. It trains a separate network that predicts the return as a Gaussian (a mean and a variance) and uses it to steer the transformer's action training. With this change it raises the Hopper-Medium score from 67.5 to 96.9 on D4RL. That critic is a full classical deep network.

Our claim has two parts, and both are falsifiable. First, the critic needs far less capacity than CGDT gives it. A critic with tens to a few hundred parameters, reading a compressed view of the transformer's own hidden state, should recover most of CGDT's gain. Second, and this is the open bet, a variational quantum critic at that budget either beats parameter-matched classical critics or it does not. The modular test bed is built so that either outcome is reported cleanly.

The idea differs from prior hybrid RL in three ways. Variational Quantum Soft Actor-Critic shrinks the policy to 41 parameters on Pendulum but keeps classical critics. Quantum Advantage Actor-Critic uses circuits for both actor and critic, but on CartPole and online, with no transformer. QTRL avoids quantum inference by having a circuit generate classical MLP weights. None of them place the circuit in a training-only role next to a transformer. None of them test the circuit against matched classical stand-ins inside the same slot.

Research Program

The critic reads the Decision Transformer's hidden vector $h_t$ at timestep $t$. A trainable linear map $P$ compresses that vector to $n$ features, with $n$ between 6 and 10, one feature per qubit. Those features are loaded repeatedly into an $L$-layer circuit using data re-uploading (feeding the inputs in again between trainable layers). Variational Quantum Soft Actor-Critic found re-uploading mandatory for learning. A small classical readout then turns the measured qubit expectations into the Gaussian return prediction:

$$ (\mu_t, \log\sigma_t) = W\,\langle Z \rangle\big(U_\theta(P h_t)\big) + b, $$

The symbols in this equation are as follows. $U_\theta$ is the circuit with trainable gate angles $\theta$. $\langle Z\rangle$ is the vector of $n$ single-qubit measurement averages. $W$ and $b$ form the readout, which outputs a mean $\mu_t$ and log standard deviation $\log\sigma_t$ of return.

The gates follow the rotation-plus-CNOT layout that Quantum Advantage Actor-Critic found trainable, using its arctan normalization to keep angles bounded. Training otherwise copies CGDT's procedure: an asymmetric negative log-likelihood loss for the critic, and expectile-guided action training for the transformer. At deployment only the transformer runs.

This setup is a classical-to-quantum transfer problem with one twist. The classical front-end feeding the circuit is the transformer currently being trained, not a frozen pretrained extractor. Quantum integration in swin transformer mitigates overfitting in breast cancer screening and Classical-to-quantum convolutional neural network transfer learning both feed static features into a circuit. We compare three ways of training the compression $P$:

  • frozen at random,
  • trained only on the critic loss,
  • trained jointly with gradients flowing back into the transformer.

The comparison tests whether representations shaped by the circuit matter.

The software is part of the method, not packaging. Altmann et al. argue that when classical feature extraction is not cleanly separated from the quantum head, the classical layers can mask what the quantum part contributes. We therefore define a single value-head contract: a batch of hidden vectors in, a mean and variance out, a declared parameter count, and a declared cost per forward pass. Every critic variant implements that contract. The data pipeline, transformer, CGDT loss, tuning sweep, and evaluation loop are shared and never touched.

This follows the decoupling that RL4LMs applied to language-model RL. It is the same discipline that DeepDIVA and FaceX-Zoo used to isolate one component of a system so it can be changed and measured alone. Quantum circuits are simulated in TorchQuantum so gradients pass through the whole system. With at most 10 qubits, the simulator tracks a state of at most 1,024 amplitudes. Batched GPU simulation then stays well short of the 14-qubit, 12-minutes-per-run regime documented in Benchmarking Quantum Reinforcement Learning.

Because the critic only shapes a training signal, it is also the most natural place to test hardware imperfection. We add simulated shot noise (finite measurement samples) and gate noise to the critic's outputs. We then measure how much the deployed policy degrades. If it degrades little, training-only placement is a practical route to using noisy near-term devices.

Baselines

The naive reference is the plain Decision Transformer with no critic, trained in our harness with the same architecture and data. Its reported score on Hopper-Medium is 67.5 (Critic-Guided Decision Transformer). Every critic variant must clear it to show that a small critic carries any signal.

The strongest existing method is CGDT with its full classical critic, reported at 96.9 on Hopper-Medium in the same paper. We rerun it in our harness so all variants share the transformer, data, and tuning budget. We reproduce rather than quote, because the comparison must be within one codebase. If our CGDT reproduction falls well short of 96.9, we report that before comparing anything else. A small critic "beats" this baseline in the sense that matters here if it recovers most of CGDT's gain over DT with one to two orders of magnitude fewer critic parameters. It need not exceed 96.9.

The decisive competitor is a set of parameter-matched classical critics inside the same interface. For each quantum budget there are three: a small MLP, a matrix-product-state tensor network (the trainable classical front-end used by Samuel Yen-Chi Chen et al.), and a linear value head on the transformer's own hidden state. Each gets the same input compression and an identical hyperparameter search of fixed size. The quantum critic beats them only if its score exceeds theirs at equal parameter count across seeds, with a bootstrap confidence interval excluding zero. A second condition applies: the margin must not shrink to nothing when the classical variant is given an equal wall-clock budget instead.

Three ablations target the quantum ingredient. The first removes all CNOT gates, so the circuit cannot create entanglement. The second freezes the circuit at random angles and trains only the readout, which tests whether the quantum layer is just a fixed random-feature map. The third deletes the classical readout to replicate the pure-circuit failure in Quantum Advantage Actor-Critic. Together they locate where any benefit comes from.

Why not just…?

Why not just use a tiny classical critic? This is the real threat. Nothing in the file shows a quantum module beating a carefully tuned classical module of equal size on a transformer-based task. The parameter-efficiency results in Benchmarking Quantum Reinforcement Learning come with the warning that apparent superiority depends on how well the classical baseline is tuned. QTRL's MiniGrid result (0.900 versus 0.916) shows parity, not superiority. The proposal is therefore designed so this alternative is a baseline, not a strawman. If matched MLP or tensor-network critics equal the circuit, we report a clean result: CGDT's critic can be shrunk drastically with no need for quantum hardware.

Why not just share the transformer and put a linear value head on it? That is the cheapest possible critic, and it is included as a baseline. Value-loss gradients flowing into a shared trunk are a known source of interference with policy learning. Our joint-training condition for $P$ directly measures whether critic gradients help or hurt the transformer. The frozen-trunk condition shows what a decoupled head buys.

Why not just use a Quantum-Train-style generator to produce critic weights? QTRL reports parameter cuts of roughly a quarter to a half on CartPole and MiniGrid. A classical low-rank or hypernetwork generator could plausibly match those, so the approach does not isolate a quantum contribution any better. More to the point, our critic is already training-only, so QTRL's main advantage of avoiding quantum inference is moot here. We keep the generator as a secondary interface implementation, not the main bet.

Why not just drop the critic and use behavior cloning or a plain Decision Transformer? The file reports the cost: DT reaches 67.5 versus CGDT's 96.9 on Hopper-Medium. DT and behavior cloning also fail on the Bernoulli Bandit, where CGDT is near-optimal. The critic is needed. The open question is only how large it must be.

Proposed Experiments

Main experiment. The primary benchmark is D4RL Hopper-Medium, the one setting with reported numbers for both DT and CGDT. We extend to the other D4RL locomotion datasets and to Antmaze, which requires stitching. For Antmaze there is no reported result found on Lacuna for CGDT, so our own reproduction sets the reference.

Luck-isolation check. The Bernoulli Bandit isolates luck from skill. It is the cleanest check that a tiny critic still separates expected return from noise. CGDT is reported "near-optimal" there, with no numeric score in the file.

Statistical protocol. Following Benchmarking Quantum Reinforcement Learning, we treat seed variance as central. Each configuration runs over many independent seeds. We report the fraction of seeds exceeding a fixed normalized-score threshold, not just mean score, with bootstrap intervals. We also report critic parameter count, critic training wall-clock, and peak memory beside every score.

Parameter sweep. We sweep qubit count from 6 to 10 and circuit depth from 1 to 4 layers. Each point is matched to classical critics of equal size. This yields score-versus-parameter curves for each critic family. The question the curves answer is whether the classical curve flattens earlier than the quantum one, as the BeamManagement6G study observed, or whether the two curves coincide.

Decisive checks. Three outcomes decide the proposal:

  • If parameter-matched classical critics equal the quantum critic, the quantum claim fails, and the result becomes a critic-compression finding.
  • If the random-angle circuit equals the trained one, any gain comes from random features, not learned quantum structure.
  • If the entanglement-free circuit equals the full one, entanglement is not the source of any benefit.

Only if the full circuit beats all three, and then degrades gracefully under simulated shot and gate noise, would we claim that training-only quantum critics are a sound way to bring near-term quantum circuits into transformer reinforcement learning.