Flow controllers that blow or suck air through jets on a body can cut drag substantially, but the jets themselves cost energy. Wind-tunnel studies show that the setting with the most drag reduction is often a net energy loser. On a high-drag Ahmed car model, running all four blowing arrays at full rate gave 29% drag reduction at an efficiency (power saved over power spent) of only 0.36, while gentler combined blowing gave 18% at an efficiency of 4.4 (Zhang et al. 2018). On the low-drag variant, an ant-colony search found a sparse 10% strategy that was 200 times more efficient than its 18% optimum (Zhang and Fan 2023). Reinforcement learning (RL) controllers for flow (Fan et al. 2020, Vinuesa et al. 2024, FluidGym) instead fold drag, lift, and any actuation penalty into one learned scalar signal. They model actuators as continuously adjustable velocities with no cost for switching a jet on.
We propose to split the reward along the line where knowledge actually differs. The fluid benefit is unknown and has to be learned or differentiated through a simulator. The actuator's energy bill is a known, closed-form function of the commanded action, including a fixed cost for each jet that is switched on. We keep a learned value estimate only for the fluid term and differentiate the actuator cost exactly. Each actuator gets an on/off gate, relaxed to the interval [0,1] and annealed toward 0 or 1 during training. Our claim is that this combination finds sparse, net-energy-positive actuation patterns that scalarized RL, L1 sparsity penalties, and post-hoc thresholding miss. Simulation budget is held equal across all methods.
Core Claim
Write the instantaneous net power saved as
$$ \Pi(s,a,z) = U_\infty\big(F_0 - F(s,a,z)\big) - \sum_{i=1}^{N} z_i\big(c_0 + \tfrac{1}{2}\rho A_i |u_i|^3\big), $$
where $U_\infty$ is the free-stream speed, $F_0$ and $F$ are the uncontrolled and controlled drag forces, $u_i$ is the jet velocity of actuator $i$ (one of $N$), $z_i\in\{0,1\}$ says whether that actuator is on, $A_i$ is its outlet area, $\rho$ is fluid density, and $c_0$ is a fixed activation cost. The fixed cost stands in for valve, compressor-idle, or plasma-ignition overhead; it is a modeling assumption that we sweep rather than fix. The first term is physics we cannot write down. The second term is known exactly.
A scalarized critic must re-learn the second term from noisy returns. This has two consequences. The cost gradient it supplies is a noisy estimate of something we could compute exactly. And the jump in cost at $z_i: 0\to1$ gets smoothed away, which is precisely the feature that makes sparse configurations optimal. Our actor ascends
$$ \nabla_{a,z}\Big[\hat{Q}_{\text{fluid}}(s,a,z)\Big] - \lambda\,\nabla_{a,z}\,P_{\text{act}}(a,z), $$
where $\hat Q_{\text{fluid}}$ is a critic trained only on the drag-power stream (or, in differentiable simulators, a direct backpropagated fluid return), $P_{\text{act}}$ is the exact actuator-cost sum above, and $\lambda$ is the price of actuation ($\lambda=1$ is physically consistent). The relaxed gates $z_i$ follow the smoothing-then-hardening schedule of Continuous Relaxation Annealing.
What is new is not "multi-critic RL for flow." The new elements are the asymmetric decomposition into a learned term and an analytic term, and the treatment of actuator selection as a relaxed combinatorial variable inside closed-loop control. Neither appears in the flow-control or relaxation papers we found.
Relation to Prior Work
The motivating evidence comes from the Ahmed-body studies. Zhang et al. 2018 mapped by hand how blowing rate trades drag reduction against efficiency across four arrays. Zhang and Fan 2023 automated the search over five jet arrays with an ant-colony algorithm, using 800 wind-tunnel runs to optimize a single cost index. Both show that the energy-optimal region is sparse and backed-off. Both optimize static open-loop settings with derivative-free search. We optimize feedback policies with gradients and make the on/off structure explicit.
Three RL flow-control papers supply the learning setup and show the gap. Fan et al. 2020 ran TD3, a standard off-policy actor-critic algorithm, on two rotating cylinders in a towing tank. Vinuesa et al. 2024 ran PPO, a standard on-policy policy-gradient algorithm, on ten spanwise actuator sets of a 3D cylinder. FluidGym provides a differentiable GPU flow solver in which direct policy control (backpropagating through the simulator) trains one to two orders of magnitude faster than PPO and SAC. All three use scalar rewards and continuous actuators with no activation cost. We reuse FluidGym as the testbed and its direct gradients as one variant of the fluid branch.
The decomposition idea comes from MultiCriticAL. That work keeps separate critic networks per objective and reports that fully separate networks beat a shared trunk, which suggests sharing reintroduces interference. Supporting evidence that dense rewards drown out small penalties comes from independent normalization in The Overthinker's DIET and separate discounting in Two-Stage Constrained Actor-Critic. We depart from these in one respect: every one of them learns every reward channel. We argue that when a channel is analytically known, learning it is pure loss, and we test that claim directly. ABDPO is the closest precedent for decomposing a physical energy. It splits molecular energy into residue-level terms with gradient surgery, which keeps its optimizer from taking degenerate shortcuts. That work is static generative design, not closed-loop control.
The relaxation method comes from Ichikawa 2024. Its Continuous Relaxation Annealing starts with a negative penalty that smooths the landscape and anneals it positive to push variables to 0 or 1. It reports that this removes the need for rounding and avoids the all-zero collapse of unannealed relaxations on dense graphs. Yamamura et al. 2023 constrain the schedule design. They show annealed soft-spin landscapes pass through a rough phase before locking in, and that stopping near the locking transition is near-optimal. We use this to guide when to freeze the gates. Both papers address static problems; we move the relaxed variables into a time-varying controlled flow.
ASRE is the main constraining result on sparsity. It argues that uniform magnitude penalties cannot separate necessary from unnecessary actions, and reports a stock-trading return of 4.721 against 0.0 for DQN and −0.584 for SAC. It is restricted to discrete actions. Our gates are its continuous, multi-actuator analogue, with the fixed cost supplying exactly the structure a uniform penalty lacks.
Research Program
We will extend FluidGym environments with a gated actuator model. In CylinderJet2D-easy-v0 we replace the paired jets with $N$ independently gated jets around the rear half of the cylinder, with $N$ from 4 to 16. In the 3D cylinder we use spanwise actuator sets, mirroring the ten-set layout of Vinuesa et al. 2024 and extending to more. In turbulent channel flow we test scaling with many wall actuators. Every environment reports the drag-power and actuator-power streams separately, computed from the expression above.
The policy outputs per-actuator amplitudes and gate logits. A gate is $z_i=\sigma(\ell_i/\tau)$, where $\ell_i$ is the logit and $\tau$ a temperature. We add a CRA-style penalty whose coefficient moves from negative, rewarding fractional gates during early exploration, to positive, forcing gates to 0 or 1. The penalty lives only in the analytic actuator branch, so annealing it does not alter the fluid critic's targets. This is the tension flagged between hardening penalties and chaotic fluid gradients, and isolating it is a deliberate design choice we test.
We will build two variants of the fluid branch. One is model-free: SAC with a critic trained only on drag power. The other is differentiable: direct policy control on the fluid return, with the actuator gradient added exactly. Gates are frozen and actuators with $z_i=0$ are physically removed before evaluation, so reported numbers come from deployable discrete hardware states.
Because the cost is analytic, a single fluid critic can also be warm-started across prices $\lambda$ and fixed costs $c_0$. This lets us trace a frontier of drag reduction against net power with far less retraining than restarting a scalar optimization for each price. The fluid critic depends on the policy, so this is a warm start, not a free lunch, and we measure how much retraining it saves.
Baselines
The do-nothing reference is the uncontrolled flow, which defines zero net power saved. Full-on uniform blowing at the amplitude that maximizes drag reduction serves as the "greedy" reference. It should reproduce the qualitative finding of Zhang et al. 2018 that peak drag reduction comes with efficiency below one.
The strongest existing competitor is scalarized net-power RL in FluidGym. This means SAC, which the FluidGym paper reports generally outperforming PPO across its environments, and direct policy control, which it reports reaching about 8% drag reduction on CylinderJet2D-easy-v0, comparable to TD-MPC and PPO (FluidGym). Each is trained on the same scalar reward $\Pi$, with the fixed cost included, and continuous actuators. We found no reported result on Lacuna for net power saved or efficiency under actuator cost in any FluidGym environment, so we will establish these numbers ourselves. Every method gets identical simulation steps, network sizes, and an equal hyperparameter search of the same number of trials, including a search over reward scaling for the scalar baselines. Beating this baseline means higher mean net power saved over at least five seeds, outside seed variability, at equal simulator budget, on at least two of the three environments.
We add two sparsity baselines on the same scalar reward: an L1 penalty on amplitudes, and post-hoc thresholding of a continuous policy with the threshold tuned on validation episodes. For the combinatorial structure, CMA-ES, a derivative-free evolutionary search, optimizes open-loop gate-and-amplitude settings. It plays the role the ant-colony search played in Zhang and Fan 2023 and gets the same simulator budget.
Three ablations isolate the ingredients. The first keeps the decomposition but uses no gates, with continuous actuators only. The second uses gates and annealing on a scalar critic, with no decomposition. The third uses MultiCriticAL-style learned critics for both streams instead of the exact cost. The third is the cleanest test of our central claim that knowing the cost beats learning it.
Why not just…?
Why not just reward net power directly and tune the scalar? In units that are physically consistent, net power is already the right scalar, so the objective is not wrong. Linear value estimates also make a summed critic formally equivalent to separate learned critics. Our argument therefore does not rest on the gradient-interference evidence from MultiCriticAL, which may not transfer. It rests on variance and smoothing. A learned critic estimates the cost gradient noisily and blurs the cost jump at activation, while our method uses the exact value. This is a real threat when $c_0$ is near zero, where we expect the gap to shrink. The scalar baseline, the learned-two-critic ablation, and a sweep over $c_0$ settle it. If the gap does not grow with $c_0$, the claim fails.
Why not just add an L1 penalty? L1 is the convex surrogate for counting active actuators, but it charges small amplitudes proportionally. A jet blowing weakly costs almost nothing under L1, yet pays the full $c_0$ physically. ASRE reports that magnitude penalties fail to separate necessary from unnecessary actions in budgeted discrete RL. We test whether the same failure holds here by measuring how many actuators L1 leaves on at low amplitude, and the resulting net power once fixed costs are charged.
Why not train continuously and threshold afterwards? Ichikawa 2024 reports that rounding unannealed relaxations degrades solutions. The Ahmed-body results suggest arrays interact non-additively, so removing one weak jet may change what the others accomplish. We measure the drop in net power between the policy before and after thresholding against our annealed policy, which needs no rounding.
Why not use derivative-free search as the ant-colony study did? For five actuators it worked (Zhang and Fan 2023). The question is how it scales. CMA-ES at equal budget, with $N$ rising from 4 to 16, shows where gradient methods pull ahead, if they do. We also report the crossover point honestly rather than only the largest $N$.
Proposed Experiments
The decisive check is ground truth on small problems. For $N=8$ in the 2D cylinder we enumerate all 256 gate subsets and optimize steady amplitudes within each by direct gradients. This gives the best open-loop net power for each subset. We then ask whether each method's final gate pattern matches, or beats through feedback, the best subset, and how often each method lands on the dense, greedy subsets instead.
The second check is the frontier. Across $\lambda$ and $c_0$ we plot drag reduction against efficiency. We expect, following the Ahmed-body results, a sparse high-efficiency branch at moderate drag reduction. The prediction is that our method reaches this branch while scalarized SAC and direct policy control cluster near the dense, high-drag-reduction end, and that the warm-started frontier costs a fraction of the simulation needed for per-price retraining.
The third check is scaling and the annealing schedule. On the 3D cylinder and channel flow we vary $N$ and the time at which gates are frozen. We test whether freezing near the point where gate values stop changing, by analogy with the rigidity transition of Yamamura et al. 2023, beats fixed schedules.
Throughout, we report net power saved, efficiency, drag reduction, lift fluctuation, number of active actuators, and simulator steps. The note succeeds if the exact-cost ablation and the $c_0$ sweep show the gains come from pricing actuation exactly, not merely from adding more networks.
References
- Active drag reduction of a high-drag Ahmed body based on steady blowing
- Artificial intelligence control of a low-drag Ahmed body using distributed jet arrays
- Reinforcement Learning for Active Flow Control in Experiments
- Towards Active Flow Control Strategies Through Deep Reinforcement Learning
- Plug-and-Play Benchmarking of Reinforcement Learning Algorithms for Large-Scale Flow Control
- Multi-Critic Actor Learning: Teaching RL Policies to Act with Style
- The Overthinker's DIET: Cutting Token Calories with DIfficulty-AwarE Training
- Two-Stage Constrained Actor-Critic for Short Video Recommendation
- Antigen-Specific Antibody Design via Direct Energy-based Preference Optimization
- Controlling Continuous Relaxation for Combinatorial Optimization
- Geometric Landscape Annealing as an Optimization Principle Underlying the Coherent Ising Machine
- Reinforcement Learning With Sparse-Executing Action via Sparsity Regularization