Reinforcement Learning versus PDE Backstepping and PI Control for\n Congested Freeway Traffic

Huan Yu, Saehong Park, Alexandre M. Bayen, Scott Moura, Miroslav Krstić

TL;DR

This paper evaluates whether model-free Reinforcement Learning (RL) can outperform traditional, mathematically rigorous PDE-based controllers in mitigating freeway traffic congestion. While RL nearly matches the performance of "perfect" model-based controllers, it proves significantly more robust when traffic conditions deviate from theoretical assumptions, though it lacks the formal safety guarantees of classical control.

What Problem Does This Paper Address?

Freeway traffic isn't just a nuisance; it’s a complex dynamical system. Specifically, "stop-and-go" waves—oscillations in traffic density and velocity—increase fuel consumption, emissions, and accident risks.

Engineers typically manage this using macroscopic models like the Aw-Rascle-Zhang (ARZ) model, which describes traffic using a system of two coupled Partial Differential Equations (PDEs). Controlling these PDEs usually involves sophisticated techniques like PDE backstepping or PI control. However, these methods have two major "real-world" flaws:

  1. Linearization: They are often designed for linearized versions of the equations, failing when traffic perturbations are large.
  2. Brittleness: They rely on precise model parameters (like the exact steady-state density). If the real-world traffic is slightly lighter or denser than the model assumes, these "optimal" controllers can actually make congestion worse.

The authors ask: Can an RL agent, which learns by trial and error without knowing the underlying physics, provide a more flexible and robust solution?

Methods

The researchers set up a "head-to-head" battle between three traditional Lyapunov-based controllers and an RL agent:

  • PDE Backstepping: A full-state feedback method that transforms the complex PDE into a simpler, stable target system.
  • Proportional (P) & Proportional-Integral (PI) Control: Standard feedback loops applied to the boundaries (on-ramps and off-ramps).
  • Reinforcement Learning (PPO): The authors used Proximal Policy Optimization (PPO), a state-of-the-art policy gradient algorithm.

The RL agent was trained in a numerical simulator of the nonlinear ARZ equations. Its "reward" was based on the $L^2$ spatial norm, essentially penalizing any deviation from a smooth, uniform flow of traffic. The control actions were implemented as "boundary control"—adjusting the flow rate at the inlet and outlet of a 500-meter freeway segment.

Datasets

The study did not use a static dataset but rather a numerical simulator based on the ARZ PDE model, discretized using the second-order Lax-Wendroff scheme.

  • Training: The RL agent underwent approximately 1,000 iterations (episodes) of training.
  • Scenarios: The agents were tested under "Perfect Knowledge" (where the controller knows the exact traffic parameters) and "Partial Knowledge" (where the actual traffic density differs from what the controller was designed for).

Key Insights and Ideas

  • Emergent Physics: Without being shown the math, the RL agent learned a policy that closely mimicked the control trajectories of the sophisticated PDE backstepping method. It "discovered" the physics of traffic stabilization through interaction.
  • The Robustness Gap: This is the "aha" moment. When the steady-state traffic density was changed (simulating real-world uncertainty), the traditional backstepping and P-controllers failed spectacularly, sometimes exacerbating waves. The RL agent, having been trained in a stochastic environment with varying densities, maintained stability.
  • The Safety Trade-off: While RL is more adaptable, it is "data-hungry" and lacks the stability guarantees of Lyapunov theory. The authors emphasize that RL is not a "safe substitute" yet because the training process involves thousands of "crashes" (numerical instabilities) that cannot happen in real life.

Key Results

  • Performance Recovery: In perfect conditions, RL achieved a cumulative reward of -104.9, nearly matching the -81.7 of the mathematically optimal backstepping controller.
  • Superior Robustness: In a "lighter traffic" scenario where the model-based controller had incorrect assumptions, RL achieved a reward of -534.3, while backstepping plummeted to -3093.3.
  • Driver Comfort: RL improved a "comfort index" (based on acceleration variance) by roughly 48% over the uncontrolled baseline, outperforming the backstepping improvement of 31%.
  • Efficiency: RL-controlled systems showed better fuel consumption metrics than their model-based counterparts in several scenarios.

Limitations and Future Work

The authors are careful to note that RL isn't ready for a "plug-and-play" deployment on the I-405 just yet:

  • Simulation-to-Real Gap: Training requires thousands of episodes. Since you can't have "test collisions" in real traffic, RL must be trained in high-fidelity simulations, which brings back the problem of model accuracy.
  • Lack of Guarantees: Unlike PDE backstepping, which mathematically guarantees the system will stabilize in finite time, RL offers no such proof.
  • Scaling: This study looked at a single 500m segment. Future work needs to address multi-segment networks with complex merging and diverging lanes.

The researchers suggest a hybrid future: using PDE theory to provide a "safety envelope" while letting RL optimize performance within those bounds.

Summary updated

Abstract

We develop reinforcement learning (RL) boundary controllers to mitigate\nstop-and-go traffic congestion on a freeway segment. The traffic dynamics of\nthe freeway segment are governed by a macroscopic Aw-Rascle-Zhang (ARZ) model,\nconsisting of $2\\times 2$ quasi-linear partial differential equations (PDEs)\nfor traffic density and velocity. Boundary stabilization of the linearized ARZ\nPDE model has been solved by PDE backstepping, guaranteeing spatial $L^2$ norm\nregulation of the traffic state to uniform density and velocity and ensuring\nthat traffic oscillations are suppressed. Collocated Proportional (P) and\nProportional-Integral (PI) controllers also provide stability guarantees under\ncertain restricted conditions, and are always applicable as model-free control\noptions through gain tuning by trail and error, or by model-free optimization.\nAlthough these approaches are mathematically elegant, the stabilization result\nonly holds locally and is usually affected by the change of model parameters.\nTherefore, we reformulate the PDE boundary control problem as a RL problem that\npursues stabilization without knowing the system dynamics, simply by observing\nthe state values. The proximal policy optimization, a neural network-based\npolicy gradient algorithm, is employed to obtain RL controllers by interacting\nwith a numerical simulator of the ARZ PDE. Being stabilization-inspired, the RL\nstate-feedback boundary controllers are compared and evaluated against the\nrigorously stabilizing controllers in two cases: (i) in a system with perfect\nknowledge of the traffic flow dynamics, and then (ii) in one with only partial\nknowledge. We obtain RL controllers that nearly recover the performance of the\nbackstepping, P, and PI controllers with perfect knowledge and outperform them\nin some cases with partial knowledge.\n
BibTeX
@misc{yu2019reinforcement,
  author = {Huan Yu and Saehong Park and Alexandre M. Bayen and Scott Moura and Miroslav Krstić},
  title = {{Reinforcement Learning versus PDE Backstepping and PI Control for\textbackslash\{\}n Congested Freeway Traffic}},
  year = {2019},
  eprint = {1904.12957},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1904.12957},
}
Figure