A drone acting as a flying radio relay has a fixed battery and two ways to spend it. Its motors use hundreds of watts. Its radio uses a few watts, but the radio's cost depends on where the drone flies, because staying close to a ground user makes each bit cheaper to send. Offline optimal control has shown that the best plan can be counter-intuitive, for example slowing down near users. Those plans assume the channel and the air are known ahead of time. When wind and radio fading are unknown, the drone has to learn online, and every sample it learns from is paid for in battery. We propose an actor-critic learner in which three quantities update together, at the same rate, on one stream of flight data. The actor is the policy that picks speed, heading and transmit power. The critic is the network that estimates how good the current policy is. The third quantity is a Lagrange multiplier, which acts as a running "price of energy" that rises whenever the drone overspends its battery budget.
The claim has two parts. On the theory side, a fully single-timescale primal-dual actor-critic can keep the near-optimal sample efficiency already proven for unconstrained single-timescale methods. The only existing finite-time constrained analysis needs three separate update speeds and pays a worse rate. On the application side, a multiplier that updates at the same speed as the policy reacts quickly when wind suddenly makes energy more expensive. The standard design, a multiplier that moves slowest of the three, keeps overspending for longer after such a change. Both parts can be tested in a physics-based relay simulator built from published fixed-wing propulsion and channel models.
Core Claim
Write the mission as a constrained Markov decision process: the drone maximizes a long-run communication reward $J_r(\theta)$ while keeping its long-run energy cost $J_c(\theta)$ below a budget $b$. Here $\theta$ are the actor's parameters, and the cost is propulsion plus radio power. The Lagrangian
$$\mathcal{L}(\theta,\lambda)=J_r(\theta)-\lambda\,(J_c(\theta)-b),\qquad \lambda\in[0,\Lambda],$$
turns the problem into a saddle point. The proposed algorithm updates the actor with step size $\alpha$, the critic weights $w$ with step size $\beta$, and the multiplier $\lambda$ with step size $\eta$ (by projected ascent on the measured overspend $\hat c_t-b$). All three are applied in the same iteration, and the ratios $\alpha/\beta$ and $\eta/\beta$ are held at fixed positive constants. They do not decay to zero.
The key observation is that the multiplier enters the reward linearly, as $r_t-\lambda_t c_t$. The critic therefore sees a reward that keeps changing, which is exactly the setting of the evolving-reward analysis. The difference is that here the drift is not set from outside. It is driven by how much the policy violates the constraint, so the drift and the actor's progress feed back into each other. The theoretical target is twice $\widetilde{\mathcal{O}}(T^{-1/2})$: one bound for stationarity of the Lagrangian in $\theta$ and one for average constraint violation. That corresponds to $\widetilde{\mathcal{O}}(\epsilon^{-2})$ samples, against the $\widetilde{\mathcal{O}}(\epsilon^{-2.5})$ of the three-timescale result.
Where this sits
The single-timescale literature supplies the method. Each of the following works proves finite-time convergence for simultaneous actor-critic updates, and none of them handles constraints:
- Finite-Time Analysis of Single-Timescale Actor-Critic treats the errors of the average-reward estimator, the critic and the actor as one coupled dynamical system, and obtains $\widetilde{\mathcal{O}}(\epsilon^{-2})$ samples with linear critics.
- On the Convergence of Continuous Single-timescale Actor-critic builds a single Lyapunov function that weighs actor and critic errors, covering continuous states and actions under Markovian sampling.
- Finite-Time Analysis of Actor-Critic Methods with Deep Neural Network Approximation extends the $\widetilde{\mathcal{O}}(T^{-1/2})$ rate to overparameterized deep networks and measures a log-log convergence slope of −0.51 on Pendulum.
We add a third coupled error, the multiplier error, to that interconnected-system argument.
Finite-time Convergence Analysis of Actor-Critic with Evolving Reward shows that the rate survives a reward that drifts at $\mathcal{O}(1/T)$. We reuse its distribution-mismatch analysis. The departure is that our drift depends on the policy and is not slow by assumption, so the proof must use the projection onto $[0,\Lambda]$ and the sign structure of the overspend to bound it.
The work that constrains our design, and that we must beat, is Finite-Time Analysis of Three-Timescale Constrained Actor-Critic and Constrained Natural Actor-Critic Algorithms.. It gives the only finite-time constrained actor-critic analysis, at $\widetilde{\mathcal{O}}(\epsilon^{-2.5})$, with linear critics and three separated update speeds, evaluated on Safety-Gym. Its separation of speeds is exactly what we remove. If the separation turns out to be necessary for stability in practice, that is a real negative result, and the experiments are designed so it would show up.
On the drone side:
- Joint Optimization of Transmission and Propulsion in UAV-Assisted Communication Networks supplies the physics and the oracle. It models a 3 kg fixed-wing drone flying at 12–28 m/s with parasitic and lift-induced drag and a Shannon/outage channel. It solves the problem offline with IPOPT and reports up to 70% radio-energy and 36% total-energy savings over constant-speed flight. We keep its model and drop its assumption of known channels.
- Mozaffari et al. motivates making propulsion part of the cost: models that omit it are invalid, because it outweighs radio power by orders of magnitude.
Two learning-based neighbours are close but differ in kind:
- Towards Scalable Intelligence: A Low-Complexity Multi-Agent Soft Actor–Critic for Large-Model-Driven UAV Swarms uses replay-buffer soft actor-critic with a scalar delay-energy utility and abstract energy terms. It has no constraint handling and no convergence analysis.
- Constrained Reinforcement Learning Using Distributional Representation for Trustworthy Quadrotor UAV Tracking Control uses constrained RL only to estimate wind disturbances for a model predictive controller, and it targets tracking error, not energy.
Energy Consumption Reduction for UAV Trajectory Training: A Transfer Learning Approach motivates our "energy spent while learning" metric. It shows that training energy matters and cuts it with transfer learning, but it uses a discrete Q-learner, which cannot output continuous speed and power commands.
Research Program
Theory. We will prove the finite-time bound for the single-timescale primal-dual method in two stages. First, with linear critics on a stationary environment, matching the assumptions of the three-timescale paper so that the rate comparison is like-for-like. Second, for overparameterized deep critics, following the deep single-timescale analysis. We do not claim a proof for non-stationary wind, which the file's Gap 3 identifies as open. For that case we state a conditional bound: the rate degrades additively in the total variation of the transition kernel. The experiments then test whether faster multiplier tracking pays off there.
One honest limitation: the constraint is on expected long-run energy, not a hard per-flight guarantee. We therefore report the fraction of individual flights that exceed the budget, alongside the average.
Simulator. We will build a relay simulator on the fixed-wing propulsion and channel model of the joint transmission-propulsion paper. To it we add Rician fading with unknown parameters, ground users who move, and a wind field that changes regime at random times. The state includes position, velocity, remaining battery, and the user channel estimates the drone can actually observe. The action is continuous acceleration, heading rate and transmit power. Data is on-policy and Markovian, matching the theory: there is no replay buffer, which the file's limitation 4 flags as outside existing proofs.
Baselines
Constant-speed flight. The naive reference is constant-speed, fixed-power flight along the nominal route, the reference against which the IPOPT paper reports its 36% total-energy saving. Every learned method should beat it on total energy at equal delivered data. It gives the budget $b$ a physical meaning: we set $b$ at 75% of constant-speed energy.
Offline IPOPT oracle. The ceiling is the offline solution of Joint Optimization of Transmission and Propulsion in UAV-Assisted Communication Networks, computed with the true channel and wind revealed in hindsight. Its reported figure, 36% total-energy saving over constant speed, is our calibration target in the static, known-channel version of the simulator. Reproducing roughly that figure confirms the simulator is faithful. Learned methods are not expected to beat the oracle. They are scored by how much of its saving they recover.
Three-timescale constrained actor-critic. The strongest existing learning competitor is C-AC from Finite-Time Analysis of Three-Timescale Constrained Actor-Critic and Constrained Natural Actor-Critic Algorithms.. Its reported results are a sample complexity of $\widetilde{\mathcal{O}}(\epsilon^{-2.5})$ and the qualitative finding that all tested algorithms met their average-cost constraints on Safety-Gym. The file gives no numeric reward or violation values for it, so its empirical bar must be set by our own reproduction. It will run with the same critic architecture, the same on-policy sample stream, and the same hyperparameter search budget, for example equal numbers of seeds and grid points over the three step sizes. Beating it means reaching the budget-feasible reward level in fewer samples, with lower cumulative overspend during learning, and without higher final violation.
Penalty-shaped and slow-multiplier ablations. Two ablations isolate the key ingredient, the speed of the multiplier, by changing nothing else in our single-timescale actor and critic. In the first, $\lambda$ is frozen at the best value found by grid search, which is the penalty-shaping approach. In the second, $\lambda$ follows a decaying slow schedule, giving a two-timescale primal-dual method. If the frozen-$\lambda$ variant matches us under shifting wind, the multiplier dynamics add nothing.
Receding-horizon MPC. The final competitor solves the same optimal control problem every few seconds using the nominal drag model and channel gains estimated online.
Why not just…?
Why not just run receding-horizon MPC with the physics model? This is the most serious threat. When the drag model is accurate and channels can be estimated, MPC may match or beat any learner, and the oracle's 36% saving shows how good model-based solutions can be. Our case rests on the unmodelled parts. The quadrotor work in Constrained Reinforcement Learning Using Distributional Representation for Trustworthy Quadrotor UAV Tracking Control reports over 70% lower tracking error once learning is added to MPC under non-zero-mean wind, which suggests nominal models fall short in exactly this regime. We test this directly by sweeping the mismatch between the true and nominal drag coefficients and wind. If MPC wins at every mismatch level, we will report that the learned method is only worthwhile when no model is available.
Why not just keep the three-timescale schedule, which is already proven? Its proven rate is worse, $\epsilon^{-2.5}$ against $\epsilon^{-2}$. On a drone, samples are battery. The structural problem is that the slowest-moving multiplier is the one that has to respond when wind raises the cost of energy. If our experiments find that same-speed multipliers make training oscillate, the three-timescale design wins on stability. The violation-curve and seed-variance measurements in the experiments would show that.
Why not just subtract a tuned energy penalty from the reward? A fixed penalty is tuned for one wind regime. When conditions change, it either overspends or wastes data rate. The evolving-reward analysis shows that slowly drifting weights are harmless, but a fixed weight provides no feedback from the budget. The frozen-$\lambda$ ablation tests this directly. If a single grid-searched penalty matches our method across wind regimes, penalty shaping is enough, and our applied claim fails.
Proposed Experiments
Rate check on Safety-Gym. To separate the algorithmic claim from simulator details, we first compare against C-AC on the Safety-Gym tasks that paper used (SafetyPointGoal1, SafetyCarGoal1, SafetyPointPush1). We plot Lagrangian-gradient norm and average violation against samples on log-log axes. The prediction is a slope near −0.5 for both, mirroring the −0.51 slope reported for unconstrained deep single-timescale actor-critic on Pendulum. C-AC should show a shallower slope. A slope clearly worse than −0.5 for our method falsifies the rate claim in practice.
Static drone simulator. We compare total energy, delivered data, the per-flight fraction over budget, and "learning energy": the battery consumed before the first budget-feasible policy, in the spirit of the training-energy metric of the transfer-learning paper. Methods are also scored on the fraction of the oracle's saving they recover. The learned policy should rediscover the slow-down-near-users behaviour, which we check by plotting speed against distance to users.
Wind regime shift: the decisive test. Halfway through training, and again during deployment, headwind strength is stepped up so that the same route costs more energy. We measure overspend accumulated after each shift and the time until the budget is met again. The prediction is that the single-timescale multiplier recovers markedly faster than both the three-timescale and slow-multiplier variants, while the frozen penalty either keeps violating or loses data rate. If recovery times are statistically indistinguishable across ten or more seeds, the same-speed multiplier gives no practical benefit, and the contribution reduces to the theoretical rate result alone.
References
- Finite-Time Analysis of Single-Timescale Actor-Critic
- On the Convergence of Continuous Single-timescale Actor-critic
- Finite-Time Analysis of Actor-Critic Methods with Deep Neural Network Approximation
- Finite-time Convergence Analysis of Actor-Critic with Evolving Reward
- Finite-Time Analysis of Three-Timescale Constrained Actor-Critic and Constrained Natural Actor-Critic Algorithms.
- Joint Optimization of Transmission and Propulsion in UAV-Assisted Communication Networks
- Efficient Deployment of Multiple Unmanned Aerial Vehicles for Optimal Wireless Coverage
- Towards Scalable Intelligence: A Low-Complexity Multi-Agent Soft Actor–Critic for Large-Model-Driven UAV Swarms
- Constrained Reinforcement Learning Using Distributional Representation for Trustworthy Quadrotor UAV Tracking Control
- Energy Consumption Reduction for UAV Trajectory Training: A Transfer Learning Approach