Abstract
This article develops a Reinforcement Learning (RL) boundary controller of stop-and-go traffic congestion on a freeway segment. The traffic dynamics are governed by a macroscopic Aw-Rascle-Zhang (ARZ) model, consisting of $2\times 2$ nonlinear Partial Differential Equations (PDEs) for traffic density and velocity. The boundary actuation of traffic flow is implemented with ramp-metering, which is a common approach to freeway congestion management. We use a discretized ARZ PDE model to describe the macroscopic freeway traffic environment for a stretch of freeway, and apply deep RL to develop continuous control on the outlet boundary. The control objective is to achieve $L^2$ norm regulation of the traffic state to a spatially uniform density and velocity. A recently developed neural network based policy gradient algorithm is employed, called proximal policy optimization. For comparison, we also consider an open-loop controller and a PDE backstepping approach. The backstepping controller is a model-based approach recently developed by the co-authors. Ultimately, we demonstrate that the RL approach nearly recovers the control performance of the model-based PDE backstepping approach, despite no a priori knowledge of the traffic flow dynamics.
Introduction
Stop-and-go traffic waves are a pervasive issue on modern freeways, leading to increased fuel consumption, higher emissions, and elevated accident risks. These phenomena are often modeled using the Aw-Rascle-Zhang (ARZ) model, a system of nonlinear Partial Differential Equations (PDEs) that characterize the evolution of traffic density ($\rho$) and velocity ($v$) over space and time. Controlling these dynamics typically involves boundary actuation, such as ramp metering or variable speed limits, to stabilize the flow toward a uniform steady state.
Traditional control strategies, such as PDE backstepping, provide rigorous stability guarantees but often rely on linearized models and precise parameter knowledge. This research investigates whether model-free Reinforcement Learning (RL) can provide a more robust alternative to these "brittle" classical methods, especially when parameters like driver relaxation time or steady-state density are uncertain.
Methodology
The study compares four control strategies: Setpoint control, PDE Backstepping, Proportional-Integral (PI) control, and Reinforcement Learning using Proximal Policy Optimization (PPO). The RL agent interacts with a discretized environment of the ARZ model, receiving rewards based on the $L^2$ spatial norm of the traffic states—essentially penalizing deviations from smooth, uniform flow.
As shown in the architecture above, the RL approach is inherently nonlinear, interacting directly with the full simulation rather than a linearized approximation. This allows the agent to discover control laws that may be better suited for the complex, non-linear nature of real-world traffic. The agent was trained over approximately 1,000 iterative episodes using synthetic data from a numerical simulator.
Results and Discussion
In "Full Knowledge" scenarios where traffic parameters are perfectly known, the RL agent nearly matched the performance of the backstepping controller, achieving a cumulative reward of -104.9 compared to backstepping's -81.7.
The true strength of RL emerged in "Partial Knowledge" scenarios. When actual traffic conditions deviated from the model's assumptions, the backstepping controller's performance plummeted (reward of -3093.3), while the RL agent remained robust (-534.4). Furthermore, RL improved the "comfort index"—a measure of acceleration variance—by 48%, significantly higher than the 31% improvement seen with backstepping. These results suggest that the RL agent effectively "learned the physics" of traffic flow and developed a policy that mimics backstepping in ideal conditions while maintaining superior stability under uncertainty.
Conclusion
While RL lacks the formal stability guarantees of Lyapunov-based control, its ability to adapt to uncertain environments makes it a powerful tool for traffic management. The study demonstrates that RL can recover the performance of sophisticated model-based controllers without a priori knowledge of the dynamics. Future work will focus on scaling these controllers to complex networks and addressing the "simulation-to-real" gap to ensure safety in real-world deployments.
Summary updated