Local Adjoints for Decentralized Flow Control

Many physical control problems are naturally split among many small actuators: jets along a cylinder, suction ports along a channel wall, heaters under a convecting fluid. Each actuator can only sense and affect its own neighborhood. Two families of methods train such controllers. Multi-agent reinforcement learning treats each actuator as an independent agent that learns by trial and error. Differentiable simulation, in which the fluid solver is written so that exact gradients can flow backward through every time step, trains a controller by direct gradient descent. Differentiable training is much faster on smooth problems, but in practice it is run as one centralized controller over the whole domain. It also has to store the entire flow history, and in turbulent flow its gradients blow up.

We propose to give each actuator agent its own gradient, computed from a local backward pass. The sensitivity of the loss to the flow state is called the adjoint. We propagate it backward only inside a small spatial ball around the agent and only over a short time window. During training, neighboring agents exchange adjoint values at their shared boundaries. At deployment they communicate nothing. The bet is that this truncation does more than save memory. In chaotic flow, the far-field and distant-future parts of the exact gradient are mostly noise. Cutting them off in space should do for distributed actuators what exponential temporal gradient decay did for drone flight: give a biased but usable descent direction where the exact gradient fails.

Core Claim

For spatially distributed actuation governed by fluid equations, a spatially and temporally truncated adjoint gives each agent a gradient that does three things. It trains as well as or better than the full centralized adjoint on smooth flows. It clearly beats the full adjoint on turbulent flows. It transfers zero-shot to larger domains. All of this should hold at memory cost that grows with neighborhood size rather than with domain size. The specific new element is the spatial truncation radius $r$, used as a credit-assignment and variance-control parameter with a predicted optimum tied to the flow's correlation length. Prior work treats truncation, if at all, only in time.

Relation to Prior Work

The closest work is the FluidGym benchmark, Plug-and-Play Benchmarking of Reinforcement Learning Algorithms for Large-Scale Flow Control. It frames distributed jets and wall actuators as a multi-agent problem and compares model-free methods against a differentiable direct policy gradient (DPC). It found DPC trains one to two orders of magnitude faster on cylinder wake control. It also found that DPC is centralized and memory-hungry, and that a multi-agent method (MA-PPO) has a slight edge in turbulent channel flow. This paper both motivates and constrains our proposal. It shows the multi-agent decomposition helps exactly where the full gradient struggles. It leaves open whether a decomposed gradient could combine the two advantages. We do not change its simulator; we change how gradients are assigned to agents.

Two papers supply our method. Learning vision-based agile flight via differentiable physics trained communication-free drone swarms by backpropagating through a shared physics loss. It showed that plain backpropagation through time fails without decaying gradients from distant time steps. We reuse that temporal decay and add its spatial counterpart. Unlike that work, we give each agent a distinct local gradient rather than one monolithic loss. torch-sla implements autograd-compatible halo exchange for domain-decomposed sparse solvers, where neighboring subdomains swap boundary values. We use the same exchange pattern to pass adjoint values between neighboring agents. torch-sla targets distributed linear algebra rather than learning agents.

Two papers frame the gap from the multi-agent side. Decentralized Safe Multi-agent Stochastic Optimal Control using Deep FBSDEs and ADMM decentralizes the forward control pass. Its backward pass, however, runs on a reduced centralized problem, and it handles only kinematic point agents. We aim for a backward pass that is local by construction, on continuum dynamics. pH-MARL builds physical energy structure into decentralized policies and reports up to twice MAPPO's reward. Its simulator is a black box. Ours is differentiated through, but our policies carry no imposed physical structure.

A constraining result comes from DaXBench. There, PPO beat differentiable methods on macro-action cloth folding because analytic gradients got trapped in local minima. We therefore restrict our claim to continuous low-level actuation, where DaXBench found differentiable methods win. We do not claim the method helps with discrete or topological exploration.

Research Program

Each agent $i$ controls actuator $a_{i,t}$ through a policy with parameters $\theta$ shared across agents. The policy sees only local sensor readings. Let $\lambda_\tau$ be the adjoint, meaning the gradient of the task loss with respect to the flow state at time $\tau$. The local gradient for agent $i$ is

$$g_i^{(r,H)}=\sum_t \big(\lambda^{(r,H)}_{i,t+1}\big)^{\top}\frac{\partial \mathcal{F}}{\partial a_{i,t}}\frac{\partial a_{i,t}}{\partial \theta}.$$

Here $\mathcal{F}$ is one solver step. $\lambda^{(r,H)}_{i,t+1}$ is the adjoint obtained by running the backward pass for at most $H$ steps. Values outside the ball $B_r(i)$ of radius $r$ cells around actuator $i$ are set to zero. The loss seeding the adjoint is the global objective restricted to $B_r(i)$, for example local wall shear stress or the agent's share of drag. The parameter update averages $g_i^{(r,H)}$ over agents. Taking $r$ as the full domain and $H$ as the full horizon recovers centralized DPC exactly. That makes the full adjoint a clean limiting case of the same code.

We will build three things. The first is a FluidGym wrapper that partitions the grid into agent-owned subdomains and runs the windowed backward pass with neighbor halo exchange. The cost is checkpointed memory of order $r^d H$ per agent, rather than the whole domain over the whole horizon.

The second is a gradient-quality analysis. At checkpoints along training we measure the cosine similarity between $g^{(r,H)}$ and the full-adjoint gradient. We also measure the variance of each across random flow initial conditions. Our prediction rests on two facts. Error from spatial truncation should shrink roughly like the decay of two-point flow correlations at distance $r$. Variance of the full adjoint should grow roughly exponentially in $H$ in chaotic regimes, because nearby trajectories separate at the leading Lyapunov rate. Together these imply a non-trivial optimum $r^*$ on the order of the flow's correlation length.

The third is a temporal-only versus spatial-only ablation, which tells us whether space adds anything beyond the known temporal decay.

Baselines

The naive reference points are zero actuation and a random policy on each task. Zero actuation defines the uncontrolled drag, Nusselt number, or wall shear against which all reductions are reported.

The strongest existing differentiable method is centralized DPC as implemented in FluidGym. On CylinderJet2D-easy-v0 it reports roughly 8% drag reduction, comparable to TD-MPC and PPO, with one to two orders of magnitude faster training than PPO and SAC (FluidGym). This is the strongest competitor because it uses the exact gradient our method approximates. We will run it with the same shared-weight local-observation policy so that only the gradient differs. On turbulent channel flow we found no reported DPC result on Lacuna, and no reported numeric MA-PPO or SAC result on Lacuna for TCF or RBC2D-hard-v0 beyond the qualitative statements that SAC is generally best and MA-PPO slightly better in TCF. We will therefore reproduce all baselines ourselves.

The strongest model-free competitors are MA-PPO and SAC from the same benchmark, run with identical policy networks and observations. Because it uses an analytic gradient through the solver, DPC with Agile Flight-style temporal decay (Learning vision-based agile flight) is the key ablation: it isolates spatial truncation.

Fairness rules apply to every method. All methods get the same simulator configuration and the same wall-clock and GPU-memory budget. The two budgets are reported separately, because differentiable methods trade compute for samples. Every method also gets the same number of hyperparameter trials (for example 20 per method per task, over learning rate and the method-specific knobs $r$, $H$, decay, and entropy coefficient), and results use at least five seeds.

Beating the baselines means two things. On CylinderJet2D, local adjoints must match DPC's drag reduction within seed noise at lower peak memory. On TCF and RBC2D, they must reach better final control than both full-adjoint DPC (with or without temporal decay) and MA-PPO under equal wall-clock.

Why not just…?

Why not just run centralized DPC with weight sharing, since it already has the exact gradient? This is the real threat. On laminar cylinder flow it may well be as good as ours, and we expect that outcome. Our claim lives in the turbulent and large-domain regimes. There, the full adjoint's memory scales with the whole domain, and its variance should grow with horizon length. The direct test is the $r$ sweep: performance against $r$, with full DPC as the right endpoint of the curve. If that curve is flat or monotone increasing on TCF, the spatial-truncation idea is wrong and we will say so.

Why not just use MA-PPO, which already wins in turbulent channel flow? FluidGym shows the multi-agent decomposition helps in TCF. Model-free training, though, pays the sample cost that differentiable methods avoid: one to two orders of magnitude on cylinder flow in the same benchmark, and a 12x interaction gap in SAM-RL on single-arm manipulation. Our method keeps MA-PPO's locality and adds analytic gradients. If MA-PPO matches us at equal wall-clock on TCF, the gradient adds nothing there.

Why not just apply temporal gradient decay, which already fixes exploding gradients? Temporal decay damps distant-future sensitivity but keeps the full spatial reach. In a wide turbulent domain, an agent's gradient then still sums noisy sensitivities from flow structures far away that it cannot influence reliably. The temporal-only versus spatial-only ablation separates the two effects. A spatial benefit has to show up beyond the best-tuned decay.

Why not just design local physical rewards, as max-pressure does for traffic? MPLight coordinates 2,510 intersections with a local pressure reward, reaching 472.51 s average travel time against 497.76 s for MaxPressure (Toward A Thousand Lights). That works because queue conservation makes local pressure an exact proxy for downstream capacity. Turbulent wall flow has no comparably clean local conserved proxy. We nonetheless include MA-PPO trained on local wall-shear rewards, which is the closest analog.

Proposed Experiments

The first experiment is a sanity check on CylinderJet2D-easy-v0. We expect local adjoints to reproduce DPC's roughly 8% drag reduction, with peak memory falling as $r$ shrinks.

The decisive experiment is on TCF and RBC2D-hard-v0. We sweep $r$ from a few cells to the full domain and $H$ from a few steps to the full horizon. For each setting we record the final reduction in wall shear stress or Nusselt number and the gradient cosine similarity and variance against the full adjoint. We then check whether the best $r$ tracks the measured correlation length when we change the Reynolds or Rayleigh number. That tracking is the mechanistic prediction that would distinguish an insight from a tuning trick.

The transfer experiment follows FluidGym's protocol. We train on a small TCF domain, deploy on larger ones without retraining, and compare transfer degradation for local-adjoint, full-adjoint, and MA-PPO policies. We expect local-adjoint policies to degrade least, because their training signal never depended on domain-wide structure.

The scaling experiment fixes the per-GPU memory budget and grows the domain until full DPC no longer fits, reporting how far local adjoints extend the trainable size.

The result we would treat as falsifying is any of three outcomes on turbulent flow: the best $r$ always equals the full domain, best-tuned temporal decay alone matches spatial truncation, or MA-PPO matches local adjoints at equal wall-clock.