When a wind turbine extracts energy, it leaves a slower, more turbulent wake behind it. Turning ("yawing") an upstream rotor slightly away from the wind sacrifices some of its own power but pushes that wake away from the turbines behind it. Learning when and how far to yaw is a sequential control problem. It is naturally solved by off-policy actor-critic reinforcement learning, which reuses past experience. In this family, a learned "critic" predicts future reward, and a "policy" picks actions that the critic rates highly. These critics are known to be systematically wrong. Maximizing over noisy predictions inflates them (overestimation), and the usual fix of taking the minimum over two critics deflates them (underestimation). Ensemble methods such as REDQ and Truncated Quantile Critics (TQC) expose a dial that moves the critic between these errors. The setting of that dial is a single global number, and the best value changes from task to task.
This proposal argues that a wind farm is a setting where one global dial is the wrong design. Front-row turbines see clean, high-energy inflow, and their yaw choices change the whole farm's power. Back-row turbines see damped, turbulent flow, and their effect on everyone else is small. We expect their critic errors to differ in size and possibly in sign. We propose a critic that is factored over the wake interaction graph, with one bias dial per turbine factor. Each dial is set online by comparing the critic's predictions with realized returns, measured only after the wake has physically traveled downstream. If this works, it gives the first off-policy, bias-controlled deep controller for wake steering. It would also show something general: bias calibration should be local wherever a value function decomposes over interacting parts with different noise levels.
Core Claim
The claim has two parts, and the first is a falsifiable precondition for the second. First, when a standard global-dial ensemble critic is trained for wake steering, the measured estimation bias of the per-turbine value factors varies systematically with a turbine's position in the wake graph. Second, setting a separate truncation level for each factor, using a delay-aware bias measurement, gives more energy per simulator step and less yaw motion than the best globally tuned dial. If the first part fails, so that per-turbine bias turns out to be uniform under the best global setting, the per-turbine machinery has no reason to exist. We would report that as the finding and keep only the delay-aware calibration of one dial.
The ingredients are borrowed, and the roles of the sources are distinct. TQC (Controlling Overestimation Bias with Truncated Mixture of Continuous Distributional Quantile Critics) pools the predicted return quantiles from several critics, drops the top $d$ most optimistic ones, and so gives a fine-grained bias dial. It supplies our per-factor target, but it tunes $d$ by hand per environment. REDQ (Randomized Ensembled Double Q-Learning: Learning Fast Without a Model) takes the minimum over a random subset of an ensemble. That keeps bias under control at high update-to-data ratios, which matters when each simulator step is expensive. Adaptive Ensemble Q-learning: Minimizing Estimation Bias via Error Feedback already tunes ensemble updates from error signals. It is the closest prior idea, and the proposal stands or falls on beating a global version of that idea. Our difference is that the feedback is per factor and is timed by wake travel.
Two prior works motivate the delay timing. Actor Critic Agents for Wind Farm Control introduced a delay-aware update based on the frozen-wake assumption, which treats the wake as a pattern carried downstream at the wind speed. That work used single linear Fourier-basis critics per turbine, with no bias control. How to craft a deep reinforcement learning policy for wind farm flow control used deep graph-attention policies, but trained them with on-policy PPO, which discards data after each update and has no critic-bias issue to manage.
Research Program
The critic is factored over a directed wake graph. Edges are computed from turbine geometry and the current wind direction, following the coordination-graph construction of Scalable Optimization for Wind Farm Control using Coordination Graphs, which used it for steady-state Thompson sampling rather than value learning. The joint value is written as
$$Q(s,a)=\sum_{i=1}^{n} Q_i\big(s_{\mathcal{N}(i)},a_{\mathcal{N}(i)}\big),$$
where $n$ is the number of turbines and $\mathcal{N}(i)$ is turbine $i$ together with its wake-graph neighbors. Each $Q_i$ is a small ensemble of quantile critics trained against a local reward: turbine $i$'s power plus a yaw-motion penalty.
The training target for factor $i$ drops the top $d_i$ pooled quantiles. The dial $d_i$ moves by a slow, signed update
$$d_i \leftarrow \mathrm{clip}\big(d_i+\eta\,\hat b_i\big),\qquad \hat b_i=\frac{1}{|B|}\sum_{t\in B}\Big(Q_i(s_t,a_t)-G_i^{(\tau_i)}(t)\Big),$$
where $\eta$ is a step size, $B$ is a window of recent transitions, and $G_i^{(\tau_i)}(t)$ is the realized discounted local return from time $t$. That return is computed only after a lag $\tau_i$, which equals the frozen-wake transit time from turbine $i$'s upstream neighbors. Positive $\hat b_i$ means the factor is overestimating, so more optimistic atoms are dropped; negative $\hat b_i$ means it is underestimating, so fewer are dropped.
The lag matters for two reasons. Without it, the target would credit the reward at time $t$ to a yaw action whose wake has not yet arrived. The bias estimate would then mix true optimism with simple timing error.
The policy is a per-turbine actor with a shared graph encoder. It outputs continuous yaw offsets, so the method also covers the continuous-yaw case that coordination-graph Thompson sampling left open. The stochastic-optimization literature supplies one analytical lens. Fast Continuous and Integer L-Shaped Heuristics Through Supervised Learning shifts neural surrogate predictions down by fixed safety buffers, so that biased surrogates do not cut away good solutions. Our $d_i$ plays the same role, but it is measured and local rather than fixed and global. Factoring the critic over a sparse graph is our analogue of decomposition: per-update cost should grow with the number of graph edges rather than with the size of the joint action space. We will report wall-clock time and memory as the number of turbines grows, to check this.
Baselines
The naive reference point is greedy alignment, with every rotor facing the wind. This is the reference against which the DFAC controller of Actor Critic Agents for Wind Farm Control reported a "45.8% increase" in power in a 3-turbine WFSim layout, along with "9.34 MWh vs. 9.25 MWh" against its DADQ comparison. We will rerun DFAC, which is linear, decentralized, and delay-aware, on the same layouts and wind sequences. Our method beats it if it captures more energy at an equal number of simulator steps. The 45.8% figure is used only as a sanity check of our reimplementation.
The strongest deep competitor is the PPO controller with a graph-attention encoder from How to craft a deep reinforcement learning policy for wind farm flow control. On a 19-turbine diamond layout with autoregressive (ARMA) wind, it reported up to "14% higher energy production" than wind tracking. Their simulator is custom, so we will rebuild the layout and wind generator as described. We will give PPO the same encoder, the same simulator-step budget, and the same hyperparameter search budget as our method. Beating it means more energy at matched steps, or equal energy in substantially fewer steps.
The strongest bias-control competitor is the same factored critic with a single global TQC dial $d$, chosen by grid search with the same tuning budget we spend on $\eta$. A second version adapts the global $d$ online using the same delay-aware feedback, in the spirit of Adaptive Ensemble Q-learning: Minimizing Estimation Bias via Error Feedback. A further version uses an unfactored REDQ critic. No reported result was found on Lacuna for REDQ, TQC, or adaptive-ensemble critics on any wind farm simulator. The only quoted numbers for these methods are MuJoCo results, such as REDQ being "roughly 3x to 8x more sample efficient than standard SAC" and TQC's "25% improvement" on Humanoid. Those numbers do not transfer as targets.
Two ablations isolate the key ingredients. The first keeps per-turbine dials but measures bias with an undelayed one-step return. The second keeps per-turbine dials but fixes each one at its row-wise grid-search optimum. The first isolates the delay timing, and the second isolates online adaptation.
Why not just…?
Why not just optimize a steady-state look-up table offline with FLORIS or PyWake? Both wind-farm reinforcement learning papers report that such tables ignore the travel time of wakes and are too rigid under fluctuating wind (How to craft a deep reinforcement learning policy for wind farm flow control). We still include a FLORIS-optimized table as a baseline under both steady and ARMA wind. Under steady wind, we expect the table to be hard to beat, and we will say so if it is.
Why not just tune one global TQC dial? This is the real threat. If the precondition fails, so that per-turbine bias is uniform, a well-tuned global dial should match our method. The precondition test comes first. We measure per-factor $\hat b_i$ under the best global $d$, grouped by wake-graph depth, before any per-turbine method is trained. The matched-budget comparison then settles the question.
Why not just use PPO, which has no off-policy critic bias at all? PPO avoids the bias problem by discarding data, and that is costly when every step of a dynamic flow simulation is expensive. The matched-step energy curves against the Kadoche et al. controller test whether off-policy reuse with bias control is worth its complexity.
Why not just add the delay correction to standard SAC with twin critics? This tests whether delay handling alone is enough, so delay-aware SAC is included. If it matches our method, the per-turbine bias contribution is not needed.
Proposed Experiments
FLORIS is used for fast development and ablation sweeps on 3- to 16-turbine layouts with time-varying wind direction. WFSim provides the dynamic evaluation, on the 3-turbine layout used by DFAC and on scaled layouts up to 32 turbines, matching the largest size reported there. The rebuilt 19-turbine diamond layout with ARMA wind gives the PPO comparison.
The metrics are total energy relative to greedy, energy versus simulator steps, cumulative absolute yaw travel as a fatigue proxy, per-factor bias against delayed Monte Carlo returns, and per-update wall-clock time and memory against turbine count. All results use ten seeds, with confidence intervals.
The decisive checks, in order, are as follows. First, does bias under the best global dial differ systematically with wake depth? Second, do per-turbine dials beat the best global dial and adaptive global dial at matched tuning budget, in both energy and yaw travel? Third, does removing the delay from the bias measurement erase the gain? This check tells whether the bias signal is physical or simply a timing artifact. Fourth, does the advantage grow from 3 to 32 turbines, as it should if more turbine positions mean more spread in bias?
References
- Controlling Overestimation Bias with Truncated Mixture of Continuous Distributional Quantile Critics
- Randomized Ensembled Double Q-Learning: Learning Fast Without a Model
- Adaptive Ensemble Q-learning: Minimizing Estimation Bias via Error Feedback
- Actor Critic Agents for Wind Farm Control
- How to craft a deep reinforcement learning policy for wind farm flow control
- Scalable Optimization for Wind Farm Control using Coordination Graphs
- Fast Continuous and Integer L-Shaped Heuristics Through Supervised Learning