Blowing air out of small slots on a body (a cylinder, a car's rear slant, a compressor blade) can keep the slow-moving layer of fluid next to the surface attached and cut drag. Where the slot sits matters as much as how hard it blows. A slot placed just upstream of the point where the flow naturally peels away can work with a small fraction of the energy needed at other positions. Every learning-based study we found finds this "sweet spot" once, offline, under a single fixed flow condition. Real flows do not hold still. When the incoming speed changes, the separation point moves along the surface, and a slot that was optimal becomes useless or even harmful.
We propose a controller for surfaces with many candidate actuators, and we split its job in two. A simple physics rule decides where to actuate: estimate the current separation point from wall measurements, then fire the slot a fixed short distance upstream. A reinforcement learning (RL) policy, meaning a controller trained by trial and error to maximize a reward, decides how hard to blow and learns a small correction to the rule. When the observed benefit departs from what the controller expects, a surprise signal re-opens exploration. The claim is that this split lets the controller follow a migrating separation point with far fewer trials than RL that searches over all actuator combinations. We also expect it to survive kinds of drift that a precomputed lookup table cannot anticipate, such as an actuator losing authority.
Core Claim and What Is New
The closest existing work, Energy-efficient flow control via optimized synthetic jet placement using deep reinforcement learning, couples an OpenFOAM simulation of flow past a cylinder at Reynolds number 100 with PPO (proximal policy optimization, a standard trial-and-error policy learner). It sweeps jet locations from 60° to 120° and shows that the natural separation point (105° on the circular cylinder, the rear corner on the square one) gives the best drag reduction per unit of injected mass. In that paper placement is an offline hyperparameter and the flow is stationary. Our proposal turns placement into an online decision that follows a moving target.
Placement is not learned from scratch. It is produced by a rule whose form is supported by several independent experiments. Synthetic jets upstream of a modal synchronization point delay transition, while jets downstream of it accelerate transition (Luo et al. (2023)). Plasma actuators closer to the natural separation line outperform ones placed on the slant (Separation Flow Control of a Generic Ground Vehicle Using an SDBD Plasma Actuator). An endwall jet works best just downstream of the separation onset line (Separation control with endwall corner jet in a high-speed compressor cascade). These results motivate the rule. They also constrain it: the right offset from the separation point has a sign and magnitude that depend on the configuration, so the offset must be a learned quantity rather than a hard-coded one.
Formally, at each control step the action is a pair. The first element is the active slot index
$$k_t = \mathrm{nearest}\big(\hat{s}_t - \delta_t\big),$$
where $\hat{s}_t$ is the separation location estimated from the sign change of wall shear (or the wall-pressure plateau), and $\delta_t$ is a scalar upstream offset. The second element is a vector of blowing intensities $u_t$ for the active slot and its neighbours. Both $\delta_t$ and $u_t$ come from a small RL policy. The reward is drag reduction minus a penalty on injected momentum.
The motivation for this split is that Active drag reduction of a high-drag Ahmed body based on steady blowing found the maximum-drag-reduction setting (29%) had power efficiency 0.36, meaning it spent more power than it saved. A balanced setting reached 18% at efficiency 4.4. Intensity is therefore where the energy trade-off lives, and it is the part worth learning.
Research Program
The hybrid structure borrows its hand-off logic from Learning to Bridge the Gap. In that work a nominal planner runs until a novelty causes failure. A learned bridge policy then takes over, and it can return control through a CallPlanner action. The authors report convergence in about 30 learning cycles on robotic tasks where pure RL failed.
We adapt the idea to a continuous field. The physics rule is the nominal controller. The learned residual on $\delta_t$ is normally kept near zero by a penalty and is allowed to grow only when a surprise signal fires. Our version differs from bridge policies in that the hand-off is continuous and spatial rather than symbolic.
The surprise signal follows the prediction-error idea of STAR and RL with Context Detection. We accumulate the error between the critic's predicted drag reduction and the realized drag reduction, and trigger adaptation past a threshold. When triggered, exploration widens through the value-dependent variance mapping of Adaptive Variance for Changing Sparse-Reward Environments. That paper proves optimal Gaussian exploration variance scales inversely with the square root of the value, and shows that agents whose variance has collapsed cannot recover after a dynamics shift.
The traffic study gives a warning: prediction-error detection breaks down when sensor noise causes aliasing, as it did at braking noise 0.3. For this reason we use wall shear sensors on both sides of each slot, so that the separation estimate itself is a second, independent change signal. We also report how often detection fails as sensor noise increases.
We deliberately do not use pathwise differentiable models. Model-Augmented Actor-Critic shows strong sample efficiency on MuJoCo but warns of compounding model error and per-step cost, and both problems are worse in vortex-shedding flows.
Baselines
The naive reference points are the uncontrolled flow and a single fixed slot pair at 105° driven by a PPO policy. The second reproduces Energy-efficient flow control via optimized synthetic jet placement using deep reinforcement learning, which reports 8% drag reduction and 99% lift suppression at 1% of inlet flow rate on the circular cylinder, and 14.4% drag reduction at 1.7% of inlet flow on the square cylinder. These are the strongest reported numbers on our primary benchmark, but they were obtained under stationary flow. We found no reported result on Lacuna for any controller on this benchmark under drifting Reynolds number. Beating this baseline means matching its stationary numbers at Re = 100 and exceeding it once the flow drifts.
The strongest competitor is an oracle gain-scheduled lookup table. We train a separate PPO policy for each of a grid of Reynolds numbers, each with its best fixed slot. At run time we switch between them using the measured inflow speed. This is what a practitioner would build. It receives the same total simulation budget as our method, split across grid points, and the same hyperparameter search.
A third baseline is flat PPO over the full slot array, with Reynolds number randomized during training and supplied as an input. This tests whether structure in placement matters at all. A fourth is model-free extremum seeking on per-slot amplitudes, in the spirit of Extremum seeking-based iterative learning linear MPC.
Three ablations isolate the key ingredients. The first replaces the physics rule with a learned slot choice. The second removes the surprise trigger, leaving fixed exploration. The third freezes $\delta_t = 0$.
All methods share the solver, mesh, sensors, actuator limits, number of simulation episodes, and tuning trials. Results are averaged over at least five seeds.
Why not just…?
Why not just use a lookup table indexed by inflow speed? This is the real threat. If drift is caused only by the measured inflow speed, a table may match or beat us. The experiments therefore include drift the table cannot see: one slot losing half its authority, and an unmeasured change of a few degrees in inflow angle. These shift the effective optimum without changing the scheduling variable. If the table equals the hybrid under these scenarios too, our claim fails, and we will report that outcome.
Why not just train flat RL over all actuators with domain randomization? The backward-facing ramp matrix in Enhancing Flow Separation Control Using Hybrid Passive and Active Actuators in a Matrix Configuration needed 120 hand-tested configurations to find a 91% pressure-recovery setting, against 43% for passive control alone. The number of slot combinations grows exponentially with array size. Flat RL can in principle find the same policy. We expect it to need many more episodes and to adapt slowly after a shift, and the recovery-time measurements will test this directly.
Why not just apply a continual-learning network such as fast/slow successor features? Balancing Plasticity and Stability with Fast and Slow Successor Features prevents policy collapse under mass drift in MuJoCo, but it requires plain SGD and slows training. We treat it as a complementary ablation: the same flat policy with consolidation. If it closes the gap, the placement structure is not what matters.
Proposed Experiments
The primary testbed extends the OpenFOAM cylinder benchmark to twelve jet-slot pairs spanning 60° to 130°. The Reynolds number drifts between 60 and 200 under three patterns: slow ramps, steps, and Ornstein–Uhlenbeck noise (a random drift that tends back toward a mean). The authority-loss and inflow-angle scenarios described above are added on top. A second testbed is a two-dimensional laminar separation bubble on a backward-facing ramp, using one streamwise column of six slots mimicking the Li and Zhang matrix, under varying inflow speed.
We measure four things:
- Drag reduction, averaged over the drift trajectory.
- Lift-fluctuation suppression.
- Injected mass-flow fraction and net power efficiency, computed as saved over spent power, following the Ahmed-body convention.
- Recovery time, defined as the number of shedding periods needed to regain 90% of pre-shift performance.
Three outcomes decide the claim. First, the hybrid must recover after a separation-shifting step at least several times faster than flat PPO, while staying within a few percent of the oracle table on in-distribution drift. Second, it must clearly beat the table on authority-loss and inflow-angle drift. Third, the square cylinder serves as a negative control: separation is pinned at the corners there, so the physics rule should add no benefit. A gain on the square cylinder would indicate that improvements elsewhere come from tuning effort rather than separation tracking.
Finally, we sweep wall-sensor noise to find where the separation estimate and the surprise trigger alias. This is the fluid analogue of the traffic study's breakdown at noise 0.3, and it tells practitioners how good the wall sensors need to be.
References
- Energy-efficient flow control via optimized synthetic jet placement using deep reinforcement learning
- Numerical Investigation of Hypersonic Flat-Plate Boundary Layer Transition Subjected to Bi-Frequency Synthetic Jet
- Separation Flow Control of a Generic Ground Vehicle Using an SDBD Plasma Actuator
- Separation control with endwall corner jet in a high-speed compressor cascade
- Active drag reduction of a high-drag Ahmed body based on steady blowing
- Learning to Bridge the Gap: Efficient Novelty Recovery with Planning and Reinforcement Learning
- A Surprise Triggered Adaptive and Reactive (STAR) Framework for Online Adaptation in Non-stationary Environments
- Reinforcement Learning based Control of Traffic Lights in Non-stationary Environments: A Case Study in a Microscopic Simulator
- Adaptive Variance for Changing Sparse-Reward Environments
- Model-Augmented Actor-Critic: Backpropagating through Paths
- Extremum seeking-based iterative learning linear MPC
- Enhancing Flow Separation Control Using Hybrid Passive and Active Actuators in a Matrix Configuration
- Balancing Plasticity and Stability with Fast and Slow Successor Features