Reinforcement Learning vs. Backstepping Control of Stop-and-Go Traffic

Huan Yu, Saehong Park, Scott Moura, Alexandre M. Bayen, Miroslav Krstić

Abstract

This article develops a Reinforcement Learning (RL) boundary controller of stop-and-go traffic congestion on a freeway segment. The traffic dynamics are governed by a macroscopic Aw-Rascle-Zhang (ARZ) model, consisting of $2\times 2$ nonlinear Partial Differential Equations (PDEs) for traffic density and velocity. The boundary actuation of traffic flow is implemented with ramp-metering, which is a common approach to freeway congestion management. We use a discretized ARZ PDE model to describe the macroscopic freeway traffic environment for a stretch of freeway, and apply deep RL to develop continuous control on the outlet boundary. The control objective is to achieve $L^2$ norm regulation of the traffic state to a spatially uniform density and velocity. A recently developed neural network based policy gradient algorithm is employed, called proximal policy optimization. For comparison, we also consider an open-loop controller and a PDE backstepping approach. The backstepping controller is a model-based approach recently developed by the co-authors. Ultimately, we demonstrate that the RL approach nearly recovers the control performance of the model-based PDE backstepping approach, despite no a priori knowledge of the traffic flow dynamics.

Introduction

Stop-and-go traffic waves are a pervasive issue on modern freeways, leading to increased fuel consumption, higher emissions, and elevated accident risks. These phenomena are often modeled using the Aw-Rascle-Zhang (ARZ) model, a system of nonlinear Partial Differential Equations (PDEs) that characterize the evolution of traffic density ($\rho$) and velocity ($v$) over space and time. Controlling these dynamics typically involves boundary actuation, such as ramp metering or variable speed limits, to stabilize the flow toward a uniform steady state.

Schematic of a freeway segment with boundary control inputs $U_{in}$ and $U_{out}$.

Traditional control strategies, such as PDE backstepping, provide rigorous stability guarantees but often rely on linearized models and precise parameter knowledge. This research investigates whether model-free Reinforcement Learning (RL) can provide a more robust alternative to these "brittle" classical methods, especially when parameters like driver relaxation time or steady-state density are uncertain.

Methodology

The study compares four control strategies: Setpoint control, PDE Backstepping, Proportional-Integral (PI) control, and Reinforcement Learning using Proximal Policy Optimization (PPO). The RL agent interacts with a discretized environment of the ARZ model, receiving rewards based on the $L^2$ spatial norm of the traffic states—essentially penalizing deviations from smooth, uniform flow.

Comparison of the model-based expert design system versus the iterative RL training process.

As shown in the architecture above, the RL approach is inherently nonlinear, interacting directly with the full simulation rather than a linearized approximation. This allows the agent to discover control laws that may be better suited for the complex, non-linear nature of real-world traffic. The agent was trained over approximately 1,000 iterative episodes using synthetic data from a numerical simulator.

Results and Discussion

In "Full Knowledge" scenarios where traffic parameters are perfectly known, the RL agent nearly matched the performance of the backstepping controller, achieving a cumulative reward of -104.9 compared to backstepping's -81.7.

The true strength of RL emerged in "Partial Knowledge" scenarios. When actual traffic conditions deviated from the model's assumptions, the backstepping controller's performance plummeted (reward of -3093.3), while the RL agent remained robust (-534.4). Furthermore, RL improved the "comfort index"—a measure of acceleration variance—by 48%, significantly higher than the 31% improvement seen with backstepping. These results suggest that the RL agent effectively "learned the physics" of traffic flow and developed a policy that mimics backstepping in ideal conditions while maintaining superior stability under uncertainty.

Conclusion

While RL lacks the formal stability guarantees of Lyapunov-based control, its ability to adapt to uncertain environments makes it a powerful tool for traffic management. The study demonstrates that RL can recover the performance of sophisticated model-based controllers without a priori knowledge of the dynamics. Future work will focus on scaling these controllers to complex networks and addressing the "simulation-to-real" gap to ensure safety in real-world deployments.

Summary updated

Abstract

This article develops a Reinforcement Learning (RL) boundary controller of stop-and-go traffic congestion on a freeway segment. The traffic dynamics are governed by a macroscopic Aw-Rascle-Zhang (ARZ) model, consisting of $2\times 2$ nonlinear Partial Differential Equations (PDEs) for traffic density and velocity. The boundary actuation of traffic flow is implemented with ramp-metering, which is a common approach to freeway congestion management. We use a discretized ARZ PDE model to describe the macroscopic freeway traffic environment for a stretch of freeway, and apply deep RL to develop continuous control on the outlet boundary. The control objective is to achieve $L^2$ norm regulation of the traffic state to a spatially uniform density and velocity. A recently developed neural network based policy gradient algorithm is employed, called proximal policy optimization. For comparison, we also consider an open-loop controller and a PDE backstepping approach. The backstepping controller is a model-based approach recently developed by the co-authors. Ultimately, we demonstrate that the RL approach nearly recovers the control performance of the model-based PDE backstepping approach, despite no \textit{a priori} knowledge of the traffic flow dynamics.
BibTeX
@misc{yu2019reinforcement,
  author = {Huan Yu and Saehong Park and Scott Moura and Alexandre M. Bayen and Miroslav Krstić},
  title = {{Reinforcement Learning vs. Backstepping Control of Stop-and-Go Traffic}},
  year = {2019},
  eprint = {1904.12957},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1904.12957},
}
Figure