Abstract
Intersections are a significant bottleneck in traffic and have been a topic of much research. Optimization approaches incorporating traffic models are often limited by the intractable complexity resulting from the combinatorial explosion associated with increasing numbers of vehicles. Learning cooperative maneuver policies with deep neural networks from traffic data is a promising approach to address this issue. This paper presents two approaches for managing traffic at intersections using deep reinforcement learning. The first approach learns an adaptive traffic signal controller, serving as a trajectory planner for all vehicles at the intersection. The second approach uses a centralized control unit to optimize efficiency and equity by ordering automated vehicles to yield to vehicles on conflicting routes with lower priorities. The approaches achieve state-of-the-art performance in terms of traffic efficiency and equity compared to non-learning and other learning-based methods.
Introduction
Urban mobility is fundamentally constrained by intersections. Traditional Traffic Signal Controllers (TSCs) typically rely on fixed-time cycles or simplistic actuated sensors, which fail to adapt to the dynamic and often unbalanced nature of real-world traffic. While Deep Reinforcement Learning (DRL) offers a path toward adaptive control, existing models often struggle with the "Efficiency-Equity Trade-off"—where maximizing total throughput can lead to the "starvation" of vehicles on minor roads—and the "Sim-to-Real Gap," where agents fail when faced with non-uniform traffic distributions.
Methodology
The proposed system utilizes Proximal Policy Optimization (PPO) enhanced by several key innovations designed to address the specific constraints of traffic management.
Adaptive Discounting and Reward Scaling
A primary challenge in traffic RL is the discrepancy between discrete simulation steps and continuous real-world time, particularly during transitional phases (yellow/all-red lights). The authors introduced Adaptive Discounting, which discounts rewards based on elapsed system time rather than RL steps. This prevents agents from exploiting transition phases to accumulate artificial rewards.
Furthermore, to stabilize training across varying traffic densities, cumulative rewards are scaled based on the intersection's current capacity. This ensures that data from high-congestion scenarios does not disproportionately bias the learning process.
The Equity Factor and CVTSC
To ensure fairness, the reward function incorporates an equity factor ($\eta$) tied to individual vehicle travel times. By penalizing the agent more heavily for every additional second a vehicle remains idle, the policy is forced to balance overall throughput with the needs of every driver.
For unsignalized intersections, the researchers developed Courteous Virtual Traffic Signal Control (CVTSC). In this paradigm, a centralized unit directs Connected Autonomous Vehicles (CAVs) to yield to human-driven vehicles (HVs) on conflicting routes, effectively using CAVs as "virtual red lights" to manage flow without physical infrastructure.
Experimental Setup
The models were trained and evaluated using the SUMO (Simulation of Urban Mobility) environment.
The study utilized two primary data sources:
- Simulated Data: Episodes ranging from 0 to 6,000 vehicles per hour with highly unbalanced lane distributions.
- Real-World Data: Traffic demand data from two major intersections in Freiburg, Germany, capturing authentic rush-hour peaks and complex layouts.
Results and Discussion
The DRL approach demonstrated significant improvements over traditional baselines, including the state-of-the-art Max-pressure controller.
- Efficiency Gains: In real-world scenarios, the proposed method reduced average travel times by approximately 30%.
- High-Density Performance: At traffic flows of 4,500–5,500 vehicles per hour, the DRL agent maintained a throughput of 86.27%, significantly higher than the 74-77% achieved by Webster’s and Uniform methods.
- Generalization: A single model trained on diverse simulated data outperformed 150 individually optimized baseline models, suggesting high robustness to varying urban conditions.
The results indicate that "fairness" is not just a social goal but a functional one; by preventing side-street queues from reaching critical mass, the system avoids the gridlock that eventually degrades main-road efficiency.
Conclusion
This research demonstrates that DRL, when augmented with adaptive discounting and equity-focused reward functions, can significantly optimize mixed-traffic intersections. Future work will focus on decentralizing this control logic through Multi-Agent RL and V2V communication, reducing the reliance on centralized infrastructure while accounting for more complex human driving behaviors.
Summary updated