In classical engineering, controlling a complex system—whether it is the air flowing over a wing or the traffic density on a freeway—requires a mathematical model. Engineers typically linearize the governing Partial Differential Equations (PDEs) to design controllers like PID or LQR. However, when systems become highly non-linear, high-dimensional, or chaotic, these traditional models break down.
A growing body of research suggests a different path: using Deep Reinforcement Learning (DRL) to discover control strategies directly from the physics of the environment. By coupling AI agents with high-fidelity simulations, researchers are uncovering non-intuitive physical mechanisms that outperform decades of human-engineered control theory.
The Closed Loop Discovery Engine
The core idea is to treat a physical simulation—such as a Computational Fluid Dynamics (CFD) solver—as a reinforcement learning environment. In this setup, an agent observes the state of the system (e.g., local pressure or velocity sensors) and takes actions (e.g., adjusting the blowing or suction of a jet).
As Steven L. Brunton notes in The transformative potential of machine learning for experiments in fluid mechanics, this creates a "renaissance" in experimental design. Instead of using ML just for post-processing data, the agent becomes an integral part of the experimental loop, learning to manipulate the flow in real-time.
This approach is "model-free," meaning the agent does not need to know the Navier-Stokes equations. It simply explores the state space and receives rewards for objectives like drag reduction or lift stabilization. Dixia Fan and George Em Karniadakis demonstrated this in Reinforcement Learning for Active Flow Control in Experiments, where an RL agent discovered optimal drag-reduction strategies for a physical cylinder in just 30 minutes of testing, matching human-designed controls without any prior knowledge of fluid mechanics.
Discovering Non-Intuitive Physics
The most compelling aspect of DRL in this field is its ability to discover "super-human" strategies—control laws that are mathematically sound but not intuitive to human designers.
- Sensor Selection: In Drag-reduction strategies in wall-bounded turbulent flows using deep reinforcement learning, Jean Rabault and Ricardo Vinuesa found that agents preferred monitoring streamwise velocity fluctuations over the wall-normal components typically used in traditional "opposition control." This suggested that longitudinal patterns provide more sensitive information for mitigating turbulence.
- Actuation Patterns: While humans often design periodic or sinusoidal signals for actuators, DRL agents often converge on complex, non-sinusoidal "double-lobed" signals. As shown in Towards Active Flow Control Strategies Through Deep Reinforcement Learning, these complex patterns allow the agent to synchronize with natural vortex shedding, stabilizing the wake more effectively than any fixed-frequency method.
- Strategic Placement: Wang Jia demonstrated in Energy-efficient flow control via optimized synthetic jet placement using deep reinforcement learning that DRL can identify the "sweet spot" for control—the natural separation point of the fluid. Placing actuators here allowed for the complete suppression of vortex shedding using only 1% of the energy required by less-optimized placements.
Stabilizing Chaos into Steady States
A recurring theme across these papers is the transition from mitigation to stabilization. Traditional controllers often try to reduce the magnitude of oscillations (like the lift fluctuations on a wing). DRL agents, however, often discover how to fundamentally alter the topology of the flow.
In Deep-reinforcement-learning-based separation control in a two-dimensional airfoil, Xavier Garcia showed that agents could suppress periodic vortex shedding entirely, forcing a chaotic wake into a stable, steady-state configuration. This led to a 43.9% reduction in drag. Similarly, in Effective control of two-dimensional Rayleigh-Bénard convection, multi-agent RL learned to force multiple convection cells to coalesce into a single structure, reaching a state of much lower heat transfer than occurs naturally.
Generalizing to Other Complex PDEs
The success of DRL in fluid dynamics is being mirrored in other systems governed by complex dynamics and PDEs, such as traffic flow and underactuated robotics.
- Traffic Flow: Huan Yu and Miroslav Krstić compared DRL to "PDE Backstepping"—a mathematically rigorous method for stabilizing stop-and-go traffic. In Reinforcement Learning Versus PDE Backstepping and PI Control for Congested Freeway Traffic, they found that while backstepping is perfect under ideal conditions, DRL is significantly more robust when the underlying physical parameters (like vehicle density) are uncertain or noisy.
- Underactuated Systems: DRL is particularly adept at "learning" the coupling between different degrees of freedom. In the case of the "ionocraft"—a micro-robot with no direct way to turn (yaw)—model-based RL discovered complex, out-of-phase oscillations to generate torque. As described in Nonholonomic Yaw Control of an Underactuated Flying Robot with Model-based Reinforcement Learning, the agent independently discovered maneuvers that usually require advanced Lie bracket mathematical theory, learning them from just 100 seconds of flight data.
Why It Works
The reason DRL succeeds where traditional control fails is its ability to internalize unmodeled dynamics. Whether it is the centripetal forces on a heavy truck (Saleh Albeaik) or the complex drag on an underwater vehicle (Yachu Hsu), DRL agents compensate for the "physics we left out" of our simplified models. By interacting directly with the high-fidelity reality (or its simulation), these agents discover the most efficient path through the system's phase space, often finding elegant solutions that human intuition had overlooked.