A radio with a finite battery faces a simple dilemma. It can spend energy now, when the channel may be poor, or save energy for a better channel later. Because data rate grows only logarithmically with transmit power, spending everything greedily wastes energy, and learned "foresighted" controllers have been shown to beat greedy ones by 15–20% in total data delivered. Current learned controllers, however, are black boxes. A reinforcement-learning agent learns a whole policy, keeps the battery budget satisfied only through reward penalties, and comes with no statement about what kind of solution it reaches. This proposal argues that almost all of the foresight such a controller needs fits into one number per device: the shadow price of energy, meaning how much future throughput one extra joule of battery is worth. Given the right price, the non-myopic problem becomes a myopic one. Each time slot, a fast greedy solver maximizes current rate minus price times power.
We therefore propose to learn only this price. A small graph network maps each device's battery level, time remaining, and local channel statistics to a price. That price feeds a fast unrolled rate-maximization solver, and the whole episode is trained end-to-end with an augmented Lagrangian on the episodic energy budgets. The claim is concrete and can be tested. If the channels over the episode are known in advance, first-order optimality (KKT stationarity) of the full episodic problem is equivalent to "per-slot stationarity of the rate solver under a constant per-device price." If channels are random, the correct price must behave like a martingale: today's price equals the expected price tomorrow whenever the battery is not empty. We can measure how far a learned controller is from this condition. This gives a stationarity certificate for non-myopic scheduling that no existing learned controller has.
Core Claim
Consider $N$ transmitter–receiver pairs with powers $p_i(t)$, rates $R(p(t);H(t))$ that depend on the channel matrix $H(t)$, and per-device episodic budgets $\sum_{t=1}^{T} p_i(t) \le B_i$. Pricing each budget with a multiplier $\lambda_i$ turns the problem into per-slot subproblems:
$$ p(t) \in \arg\max_{p}\; R(p; H(t)) - \sum_i \lambda_i(t)\, p_i . $$
Each subproblem is a weighted-MMSE-style power control with a linear energy cost, and non-convex local methods only reach its stationary points. Our first claim is exact and elementary. With full channel knowledge, a sequence of per-slot stationary points computed under a constant price $\lambda_i$ that satisfies complementary slackness is a KKT point of the episodic problem, and every episodic KKT point has this form. Non-myopic behavior is therefore greedy behavior at the right price. "Strategic waiting" is simply the case where the price exceeds the marginal rate available on a bad channel.
Our second claim concerns the online, random-channel case. The price becomes a function of the device's state, and first-order optimality becomes an Euler-type condition: $\lambda_i(t) = \mathbb{E}[\lambda_i(t+1)\mid \text{state}_t]$ while the battery is non-empty. The Euler residual, the gap between the two sides of that equation, is a measurable surrogate for stationarity of the dynamic problem.
Our third claim is empirical. Learning only this scalar price per device, trained by differentiating through the episode under an augmented Lagrangian, should match or beat full-policy reinforcement learning on sum-rate. It should also generalize better across episode lengths and channel shifts, and its gap to the known-channel oracle should be directly measurable.
Relation to Prior Work
The closest work is Learning Non-myopic Power Allocation in Constrained Scenarios (NMPA). It wraps an unfolded weighted-MMSE solver in a graph network that outputs a per-node scaling factor in $[0,1]$, trains that network with TD3 (an off-policy actor-critic RL method), and adds reward penalties for spending from an empty battery. It reports 15–20% episodic sum-rate gains over myopic allocation, near-zero violations, and about 7 ms inference. NMPA motivates this proposal and supplies its test bed, and we differ from it on three axes.
First, we modulate the solver through a price, not a multiplicative scale. A scale shrinks all links proportionally. A price changes which links the solver favors under interference, and it is the object that optimality theory says should carry the foresight.
Second, we train by pathwise differentiation through the episode with an augmented Lagrangian, not by model-free RL with shaped rewards. This is possible because channels are exogenous and battery dynamics are deterministic given the chosen powers.
Third, we measure stationarity: the per-slot KKT residual, the gap to the known-channel oracle, and the Euler residual. NMPA reports only sum-rate and violations.
The inner solver comes from Unfolding WMMSE Using Graph Neural Networks for Efficient Power Allocation. It reaches a sum-rate of 83.21 versus 82.94 for classical WMMSE on 20 pairs, using 4 unrolled layers instead of about 100 iterations, at roughly 2 ms per sample. We retrain it with a price input.
Lagrangian decoupling of a time- or arm-coupled budget has precedent. Relaxed Indexability and Index Policy for Partially Observable Restless Bandits relaxes per-slot activation limits into an averaged constraint whose multiplier yields approximate Whittle indices, where each index is a per-arm price for activating that arm. We adopt the same philosophy that "a price decouples the problem." We apply it to continuous power under interference, where no index structure exists and the per-slot problem is non-convex.
Combining Lyapunov Optimization With Evolutionary Transfer Optimization for Long-Term Energy Minimization in IRS-Aided Communications (LETO) uses drift-plus-penalty. That method reduces a long-term problem to per-slot problems weighted by virtual queues, which act as a hand-designed, state-dependent price. It solves each slot with differential evolution in 14–18 ms and keeps queues at 0.6–1.0 MB where myopic control diverges. This is a genuine competitor, discussed below.
The training method borrows from two augmented-Lagrangian papers. AL-CoLe: Augmented Lagrangian for Constrained Learning shows in static supervised learning that an augmented Lagrangian with increasing, shifted penalties removes dual oscillation and gives last-iterate feasibility; for example, it cuts the COMPAS flip rate from 7.6% to 1.5%. Augmented Proximal Policy Optimization for Safe Reinforcement Learning shows the same smoothing effect in constrained RL and reports 41% higher reward on Car Button. Both papers assume, in theory, that the primal step is solved well. The universal-class analysis in Constrained Learning with Universally Learnable Concept Classes makes the same global-minimizer assumption explicitly.
Two papers constrain what we can promise. Wangmeng Zuo et al. show that exact per-coordinate optimization can still stall at a joint stationary point, and Renbo Zhao et al. show that first-order methods cannot tell minima from saddles. We therefore claim only first-order stationarity, and we quantify the remaining gap empirically against multi-start oracles.
Boundary-Seeking Policy Gradient for Safe Reinforcement Learning argues that constrained optima sit on the constraint boundary, and it proves $O(1/\sqrt{T})$ residual convergence for a single constraint. Our complementary slackness check is the multi-constraint analogue. It is also more careful: under interference, a device's budget may legitimately stay slack, so boundary-seeking is not always right.
One tension in the literature largely dissolves in our setting. The concern that the augmented quadratic term couples agents does not apply to per-device budgets, because each budget involves one device. Its penalty $\tfrac{\rho}{2}(\sum_t p_i(t)-B_i)_+^2$, with $\rho$ the penalty weight and $(\cdot)_+$ keeping only overspend, couples time but not devices. That time coupling is harmless under end-to-end episode training. Cross-device coupling arises only for shared budgets, such as a common harvester, which we treat as an extension.
Research Program
We will build three components.
The first is a priced unrolled solver. We extend the unfolded weighted-MMSE solver with a per-node linear power cost, which changes each layer's closed-form power update by a shift in its denominator. We train it to return per-slot stationary points across a range of prices and verify its per-slot KKT residual against converged classical WMMSE with the same cost.
The second is a known-channel oracle. For each test episode we compute an episodic KKT point by bisection on each device's constant price, with classical WMMSE run to convergence inside, restarted from several initial points to expose local-optimum spread. This yields an attainable reference and, per claim one, the "correct" constant prices.
The third is a price network. A graph network outputs $\lambda_i(t)$ from each device's battery, time remaining, and a summary of its recent channel. It is trained by differentiating episodic sum-rate through the priced solver and the battery dynamics, with an augmented Lagrangian on each device's episodic budget: multipliers are updated by accumulated overspend and $\rho$ is increased with shifts in the AL-CoLe style. At deployment, powers are additionally clipped to the remaining battery, so no device can ever overspend. Training does not rely on the clip, because clipping kills gradients.
Analytically, we will prove the known-channel equivalence in claim one. We will then bound the episodic KKT residual in terms of the per-slot stationarity error $\epsilon$ and the price's deviation from complementarity. We expect a residual of order $\epsilon\sqrt{T}$ plus the deviation, because per-slot residuals add up over $T$ roughly independent slots. We will also state the Euler condition for the random-channel case. We do not claim global optimality.
Baselines
The naive reference is myopic unrolled WMMSE, which spends each slot as if the battery were unlimited until it runs out. It is also run with a fixed per-slot spending cap of $B_i/T$ and with a tuned safety margin.
The strongest existing learned method is NMPA (paper): TD3 with graph-network scaling and penalty shaping. Its reported figure is a 15–20% episodic sum-rate improvement over myopic allocation, with near-zero violations and about 7 ms inference, on 10 pairs in a 60×60 area with Rayleigh fading. We will reimplement it on identical channel draws with the same unrolled inner solver, and give it the same number of environment episodes and hyperparameter trials as our method. Beating it means higher mean episodic sum-rate on held-out episodes, with non-overlapping confidence intervals across seeds, at equal or lower violation and comparable inference time. Because our reimplementation may not reproduce the reported gain, we report its gain relative to our myopic run so the comparison is internally consistent.
The Lyapunov drift-plus-penalty baseline, in the spirit of LETO, sets each device's price proportional to its battery deficit, with the tradeoff weight tuned per episode length, and uses our priced solver per slot. It isolates whether learning the price beats a principled hand-designed one. We found no reported result on Lacuna for Lyapunov control on this ad hoc interference benchmark; LETO's numbers are for IRS uplink queues and are not comparable.
A constant-price policy tunes one $\lambda$ per budget level offline. It tests whether state dependence matters at all.
The known-channel oracle is an upper reference. Online methods should not beat it, and the fraction of the gap between myopic control and the oracle that each method closes is our headline metric.
Three ablations isolate our ingredients. A pure Lagrangian version removes the quadratic term, and a pure penalty version removes the multipliers; both test the augmented-Lagrangian contribution. A scale-not-price version trains NMPA's scaling-factor parameterization with our differentiable augmented-Lagrangian training. That ablation separates what the price parameterization contributes from what the training method contributes.
Why not just…?
Why not just swap NMPA's reward penalty for an augmented-Lagrangian RL update, as APPO does? This is the most obvious move and a real threat. It would fix the penalty-shaping weakness APPO documents, namely oscillation and an overly conservative interior solution. It would not change what is learned, though: a full policy with no link to the problem's optimality conditions. We include it as a baseline, NMPA's architecture trained with an APPO-style augmented-Lagrangian objective. If it matches our method on sum-rate, oracle gap, and generalization, our contribution shrinks to the stationarity diagnostics and we will say so. We expect it to lag on generalization across episode lengths and channel shift, because it must relearn time-to-go effects that our price network sees explicitly.
Why not just use Lyapunov drift-plus-penalty, which already gives a state-dependent price with no training? Drift-plus-penalty is designed for long-run averages. Its price depends on the battery but not on time remaining, so it cannot know that energy left in the final slots is worthless. We therefore predict end-of-episode leftover energy and a sizeable oracle gap on short episodes, with the gap narrowing on long ones. If Lyapunov pricing is within noise of our method at 150 steps, we will report that learning helps only at finite horizons.
Why not just use greedy allocation with a safety margin or an even spending split? Li et al. proved that greedy allocation is optimal when there is no storage. With a battery, NMPA reports that myopic allocation loses 15–20%, and BSPG's argument predicts that margins leave throughput unharvested. We include these as the naive references. We also run a sanity check: as battery capacity shrinks toward zero, all methods should converge to greedy, and our learned price should become irrelevant.
Proposed Experiments
We use the synthetic ad hoc interference simulator from the NMPA and UWMMSE papers: 10 and 20 pairs dropped in a 60×60 area, Rayleigh fading, path-loss exponent 2.2, and uniform initial batteries. We train on 100-step episodes and test on 30 to 150 steps. We add three stress settings: a mid-episode shift in mean path loss, temporally correlated fading, and random energy harvesting arrivals.
We report five metrics: episodic sum-rate, fraction of the myopic-to-oracle gap closed, violation attempts before clipping, leftover energy at episode end, and inference time. We also report two stationarity diagnostics: the mean per-slot KKT residual of the solver, and the mean Euler residual $|\lambda_i(t)-\hat{\mathbb{E}}[\lambda_i(t+1)]|$ on non-empty batteries. The expectation in the Euler residual is estimated by resampling future channels from the same state.
The decisive checks are three. First, in the known-channel setting, our constant-price stationary solution should match the oracle's sum-rate, confirming claim one numerically. Second, online, the price network's Euler residual should be lower than that of prices implied by NMPA and Lyapunov policies. We back those prices out as the marginal rate per watt at each chosen power, and we expect lower Euler residual to track a smaller oracle gap across methods. If it does not track, the diagnostic fails and we will report that. Third, on the length and distribution-shift tests, the price network should degrade less than NMPA and the APPO-style variant.
The multi-start oracle spread bounds how much any first-order method can be blamed for local optima, as opposed to lack of foresight. This addresses the stationary-point caveats raised by Zuo et al. and Zhao et al. directly, rather than assuming them away.
References
- Learning Non-myopic Power Allocation in Constrained Scenarios
- Unfolding WMMSE Using Graph Neural Networks for Efficient Power Allocation
- Relaxed Indexability and Index Policy for Partially Observable Restless Bandits
- Combining Lyapunov Optimization With Evolutionary Transfer Optimization for Long-Term Energy Minimization in IRS-Aided Communications
- AL-CoLe: Augmented Lagrangian for Constrained Learning
- Augmented Proximal Policy Optimization for Safe Reinforcement Learning
- Constrained Learning with Universally Learnable Concept Classes
- A Generalized Iterated Shrinkage Algorithm for Non-convex Sparse Coding
- A Unified Convergence Analysis of the Multiplicative Update Algorithm for Regularized Nonnegative Matrix Factorization
- Boundary-Seeking Policy Gradient for Safe Reinforcement Learning
- Performance Analysis for Energy Harvesting Communication Systems: From Throughput to Energy Diversity