A learned beamformer is a neural network that decides how a base station weights its antennas. When one is trained with hard rules such as "never leak more than this much interference to a protected user," the standard tool is a Lagrange multiplier: a price on violating the rule that rises whenever the rule is broken. Over long deployments, two separate things go wrong. The channel estimates the network sees are noisy and go stale as users move, which is the imperfect channel state information (CSI) problem. The network itself also gradually loses its ability to learn, which is called loss of plasticity: units go permanently silent and internal features collapse onto a few directions. Our central observation is that these failures meet in the multiplier. When a network has lost plasticity, raising the price no longer changes its behavior, so the price climbs without bound. That runaway price then hits the network's few remaining working units with very large gradients.
We propose to measure directly how much the network's constraint cost responds to the price, using a quantity computable from two minibatches. We will show that this "responsiveness" separates two situations that look identical from the price alone: noise-driven price swings caused by bad CSI, and capacity-driven price runaway caused by plasticity loss. We will then use it to decide when to stop raising the price and when to recycle dead units. If this works, a safe learned beamformer can run over long mobile trajectories without either freezing or violating its interference limits. The price can be trusted again as a signal, instead of relying on the fixed thresholds and reset schedules that current plasticity fixes require.
Core Claim
Primal-dual Lagrangian optimization treats constrained learning as a game between a policy that maximizes reward minus priced cost and a multiplier $\lambda$ that increases while the constraint is violated. Its guarantees rest on the policy staying expressive. Constrained Reinforcement Learning Has Zero Duality Gap proves that the gap between the constrained problem and its price-based relaxation is bounded by the policy's approximation error $\epsilon$. Natural Policy Gradient Primal-Dual Method for Constrained Markov Decision Processes gives convergence rates that also treat that error as a fixed constant. The plasticity literature shows the constant is not fixed. The Dormant Neuron Phenomenon in Deep Reinforcement Learning finds that shifting targets permanently silence units. Understanding and Preventing Capacity Loss in Reinforcement Learning finds that feature rank collapses. Curvature Explains Loss of Plasticity finds that curvature directions disappear. Our claim is that in constrained continual learning, this growth in effective $\epsilon$ first appears as a specific, measurable signature: the price keeps rising while the cost stops responding to it. Wireless beamforming under imperfect CSI is the regime where this matters most and is hardest to detect, because noisy constraint measurements make the price swing even when the network is healthy.
The responsiveness quantity follows from one gradient step. Take a primal step $\theta' = \theta + \eta(\nabla J_R - \lambda \nabla J_C)$, where $\theta$ are the network weights, $\eta$ the step size, and $J_R$, $J_C$ the expected reward and constraint cost. To first order, the sensitivity of the resulting cost to the price is
$$\kappa_t \;=\; -\frac{\partial J_C(\theta')}{\partial \lambda} \;\approx\; \eta\, \nabla_\theta J_C^{\top} \nabla_\theta J_C .$$
This is the cost gradient's squared length as seen through the network's current tangent kernel. It shrinks exactly when the gradients that could reduce cost pass through dead units or collapsed feature directions. Under imperfect CSI the naive estimate is biased upward by estimation noise, because the squared norm of a noisy gradient includes its variance. We therefore use the split-batch estimate $\hat\kappa_t = \eta\, g_1^{\top} g_2$, where $g_1, g_2$ are cost gradients computed on two independent minibatches. This estimate is unbiased for the noise-free quantity whenever the two minibatches' channel errors are independent.
Research Program
The method, which we call responsiveness-gated primal-dual learning, adds two rules on top of an off-policy augmented Lagrangian learner. Suppose the price has risen for a window of updates while $\hat\kappa_t$ has fallen below a fraction of its running median. Then the first rule freezes the dual update, so the price stops ratcheting. The second rule recycles units in the policy and cost critic. Following the ReDo recipe from The Dormant Neuron Phenomenon in Deep Reinforcement Learning, recycling means reinitializing incoming weights and zeroing outgoing weights, which leaves the network's current output unchanged. Units are ranked by how little they contribute to the cost gradient rather than by raw activation, so recycling targets the pathway the constraint needs. Dual updates resume once $\hat\kappa_t$ recovers. The fixed dormancy threshold and reset schedule are replaced by a trigger that is tied to the constraint that is actually failing. This addresses the threshold sensitivity the plasticity literature flags as open.
The work has three stages. The diagnostic stage asks whether $\hat\kappa_t$ collapse precedes both multiplier runaway and the rise in constraint violations. It also asks whether $\hat\kappa_t$ tracks the dormant-unit fraction and feature rank. It must also show that $\hat\kappa_t$ does not collapse when only CSI noise is increased on a healthy network. The intervention stage tests whether gating and targeted recycling reduce cumulative violations at matched throughput. The analysis stage attempts a bound that replaces the constant $\epsilon$ in the duality-gap result with a term controlled by $\hat\kappa_t$. We treat this bound as a stretch goal, not a deliverable.
The primary environment is a downlink multi-user beamforming task built on the DeepMIMO ray-tracing data as configured in Normalization-Free Knowledge Distillation for Variable-Bit CSI Feedback in Massive MIMO. That configuration uses scenario O1 at 140 GHz, a 64-antenna base-station array, and 128 subcarriers. We restrict to single-antenna user terminals and a subset of subcarriers to keep control tractable. The action is a continuous precoder under a total power limit. The reward is sum rate. The constraint caps the interference received by a protected user. As the research file's zero-forcing example makes explicit, interference leakage grows with estimation error times transmit power, so this is the constraint imperfect CSI hurts most.
Users move along grid routes that cross line-of-sight and blocked regions. CSI arrives with a Gaussian error whose variance is itself varied over time, and with aging delay. Following the normalization finding in the same paper, inputs retain absolute scale rather than being standardized, so path-loss information is not stripped. To check that nothing is wireless-specific, we repeat the study on Safety Gymnasium (Safety Gymnasium: A Unified Safe Reinforcement Learning Benchmark) velocity-constrained tasks. There, dynamics parameters are shifted periodically and Gaussian noise is injected into cost observations as an analogue of CSI error.
Relation to Prior Work
Three papers supply the machinery we reuse. Off-Policy Primal-Dual Safe Reinforcement Learning introduces the conservative augmented Lagrangian (CAL). CAL damps price oscillations with a quadratic penalty and guards against cost underestimation with a pessimistic critic ensemble. We build on CAL directly. It does not monitor network capacity, and its evaluation uses stationary robotics tasks and auto-bidding. ReDo supplies the recycling operation, which we trigger and target differently. The curvature account supplies the reason a gradient-norm measure should fall as plasticity is lost.
Two papers motivate the problem without solving it. Primal-Dual Continual Learning: Stability and Plasticity through Lagrange Multipliers already reads multipliers as a diagnostic. It uses them as measures of task difficulty to allocate replay memory. However, it targets forgetting of old tasks in supervised classification, and it assumes the network has enough capacity to satisfy every constraint. That assumption is exactly what fails in our setting, and our diagnostic tests whether it holds. Multi-IRS-assisted Multi-Cell Uplink MIMO Communications under Imperfect CSI: A Deep Reinforcement Learning Approach shows that learned agents using only scalar power feedback can compete with full-CSI baselines. It uses unconstrained sum-rate DQN over discrete codebooks on stationary channels, with no constraints and no long-horizon drift.
One paper constrains the design. The InFeR regularizer from Understanding and Preventing Capacity Loss in Reinforcement Learning anchors features to their random initialization and slows learning in dense-reward games. Moving users that cross between line-of-sight and blocked channels plausibly need new features, so we treat anchoring as a baseline rather than the mechanism.
Baselines
The naive reference point is CAL alone, with no plasticity treatment, run for the full long trajectory. It is the method we extend, so every gain must be measured against it. The linear-price counterpart, a Lagrangian actor-critic without the augmented penalty, shows how much of the problem the quadratic penalty already absorbs. In Safety Gymnasium velocity tasks, PPO-Lag reports a 98% reduction in safety violations relative to PPO at a 45% drop in total reward (Safety Gymnasium). This is the only numeric reference in the file for a constrained learner on these tasks. For CAL, the file reports only qualitative claims, namely comparable performance to on-policy methods with an order of magnitude fewer samples, and no scalar values. For a constrained learned beamformer on DeepMIMO there is no reported result found on Lacuna, so our CAL runs establish the reference.
The strongest competitor is CAL combined with scheduled ReDo, with its dormancy threshold and reset period tuned by grid search on the first quarter of the trajectory. Its competitive strength comes from ReDo being the only method in The Dormant Neuron Phenomenon in Deep Reinforcement Learning that did not degrade SAC on MuJoCo, where weight decay and resets did. We also run CAL with InFeR, CAL with the Wasserstein initialization regularizer from Curvature Explains Loss of Plasticity, and CAL with a hard cap on $\lambda$.
On the wireless side, the non-learned reference is a per-slot convex precoder solved from the current estimated channel with a leakage constraint tightened by a margin sized to the error variance. It has no plasticity to lose and shows what learning must beat. A last reference, a network freshly trained at each drift event, measures the plasticity gap directly. All learned methods share architectures, replay ratio, environment steps, seeds (ten per condition), and an equal tuning budget of trials per method.
Two ablations isolate the key ingredients. One uses the gate with the price frozen but no recycling. The other uses recycling triggered by the gate but ranked by raw activation instead of cost-gradient contribution. A third variant uses the naive single-batch $\kappa$ to test whether the noise correction matters under CSI error.
The proposal beats the field if it reduces cumulative constraint violation over the whole trajectory relative to tuned scheduled ReDo plus CAL, at equal or higher mean sum rate, and if its price trajectory stays bounded after drift events where the baselines' prices run away.
Why not just…?
Why not just add scheduled ReDo to CAL? This is the real threat. If tuned scheduled recycling matches our gated version, the intervention contributes nothing beyond the combination. The contribution then shrinks to the diagnostic claim that multiplier runaway is a plasticity symptom. We commit in advance to reporting that outcome. Our reason to expect a difference is that the best reset period depends on how fast users move and how noise changes. A schedule tuned on the first quarter of a trajectory should then misfire later, recycling too early during calm stretches and too late at abrupt line-of-sight losses. The late-trajectory comparison and the ablation that swaps our ranking for activation-based ranking settle this.
Why not just cap or damp the multiplier? CAL's quadratic penalty and a hard price cap both stop runaway. If runaway is only an optimization artifact, they are enough. If the network has actually lost the ability to lower cost, capping the price trades runaway for sustained violation. The diagnostic stage distinguishes these. Under a capped price, sustained violation together with low $\hat\kappa_t$ and a high dormant fraction points to capacity loss. Violations that resolve under the cap would show that our recycling is unnecessary.
Why not just re-solve a robust convex precoder every slot? A per-slot solver cannot lose plasticity. On a modest 64-antenna array it may well win. We include it and report the throughput and violation it achieves. The case for learning rests on settings where the solver lacks an explicit channel model or cannot run within the coherence time, which the multi-IRS results suggest. It also rests on the worst-case conservatism that the file's overview attributes to such methods. If the solver dominates on our benchmark, we will say so and restrict our claims to the Safety Gymnasium transfer and to the diagnostic.
Why not just use a bigger network? Curvature Explains Loss of Plasticity reports that width and depth only delay plasticity loss, and base-station hardware budgets make large policies unattractive. We run a width sweep to check whether the delay is long enough to cover the deployment horizon in practice.
Decisive Checks
The first check is temporal precedence. Across seeds, the drop in $\hat\kappa_t$ must come before multiplier runaway and before the violation increase. It must also correlate with the dormant fraction and with feature rank. The second check is noise separation. Raising the CSI error variance on a freshly trained network must make the raw price noisier while leaving the split-batch $\hat\kappa_t$ roughly unchanged, and the single-batch estimate must fail this test. The third check is the head-to-head against tuned scheduled ReDo plus CAL on cumulative violation and sum rate, reported separately for the final half of the trajectory. The fourth check is plasticity itself: the gap between each continually trained learner and the network trained fresh at each drift event, measured as learning speed and final return. Passing the first two checks without the third still yields a publishable diagnostic. Failing the second means CSI noise and capacity loss cannot be separated from the multiplier, and the proposal's central premise is wrong.
References
- Constrained Reinforcement Learning Has Zero Duality Gap
- Natural Policy Gradient Primal-Dual Method for Constrained Markov Decision Processes
- The Dormant Neuron Phenomenon in Deep Reinforcement Learning
- Understanding and Preventing Capacity Loss in Reinforcement Learning
- Curvature Explains Loss of Plasticity
- Normalization-Free Knowledge Distillation for Variable-Bit CSI Feedback in Massive MIMO
- Safety Gymnasium: A Unified Safe Reinforcement Learning Benchmark
- Off-Policy Primal-Dual Safe Reinforcement Learning
- Primal-Dual Continual Learning: Stability and Plasticity through Lagrange Multipliers
- Multi-IRS-assisted Multi-Cell Uplink MIMO Communications under Imperfect CSI: A Deep Reinforcement Learning Approach