TL;DR
This paper introduces "multi-hop upstream pressure," a novel metric for traffic signal control that considers traffic conditions beyond immediate intersections. By integrating this farsighted metric into deep reinforcement learning agents, the authors demonstrate significant reductions in network delays and improved traffic flow, especially in congested scenarios.
What Problem Does This Paper Address?
Traffic signal control (TSC) is vital for managing urban traffic flow and minimizing delays. While adaptive methods like those using "traffic pressure" have shown promise, they often suffer from a myopic view, focusing only on immediate upstream traffic. This limited perspective leads to suboptimal decisions, especially in complex networks where congestion upstream can significantly impact downstream intersections. The authors argue that effective signal control requires a broader spatial awareness, considering traffic conditions several intersections upstream. Existing deep reinforcement learning (RL) approaches that try to achieve this often increase learning complexity. This paper aims to bridge this gap by proposing a more compact and efficient way to incorporate this broader spatial awareness into RL-based TSC.
Methods
The core innovation is the concept of multi-hop upstream pressure. This metric is derived from Markov chain theory and generalizes traditional traffic pressure by accounting for traffic conditions at multiple upstream hops, not just the immediate one. This is achieved by using powers of the traffic network's transition matrix (representing vehicle movement probabilities) to weigh the influence of distant upstream queues.
This multi-hop upstream pressure is then integrated into a deep reinforcement learning framework for traffic signal control. Specifically, it's incorporated into:
- Observation Space: The RL agent observes the multi-hop upstream pressure for each traffic phase at an intersection.
- Reward Design: The reward function is based on "multi-hop upstream potentials," which are derived from the difference between upstream and downstream potentials. This encourages agents to clear upstream queues proactively.
The authors model traffic signal control as a Decentralized Markov Decision Process (DecMDP), where each intersection is controlled by an independent agent. They employ the Proximal Policy Optimization (PPO) algorithm for training these agents.
Datasets
The paper uses both synthetic and realistic traffic scenarios for evaluation:
- Synthetic Scenarios: Two networks (1x2 and 1x3 intersections) were created with varying traffic demand levels (undersaturated, slightly saturated, heavily saturated). These scenarios were designed to test the method's performance under different congestion levels and network complexities.
- Realistic Scenario (Toronto Testbed): A real-world traffic network in Toronto, Canada, was simulated. This testbed includes 12 signalized intersections, with four critical intersections controlled by the proposed method and the rest by a city plan. The traffic demand was calibrated using publicly available data.
Key Insights and Ideas
- Beyond Myopia is Crucial: The most significant insight is that considering traffic conditions beyond immediate upstream links is not just beneficial but essential for effective traffic signal control, especially under moderate to heavy congestion.
- Multi-hop Pressure as an Efficient Proxy: The proposed multi-hop upstream pressure metric effectively captures broader spatial awareness without the computational overhead of complex agent communication networks. It provides a compact and informative signal to the RL agent.
- Potential-Based Reward Drives Proactive Clearing: The potential-based reward design, unlike pressure-based rewards that might encourage holding vehicles to minimize pressure, actively incentivizes agents to clear upstream queues, leading to smoother flow and reduced delays.
- Scalability of Benefits: The benefits of multi-hop pressure are most pronounced in saturated and heavily saturated traffic conditions, but it does not harm performance in undersaturated scenarios.
Key Results
- Significant Delay Reduction: In synthetic scenarios, the proposed "Ours: 1-hop" method consistently outperformed the pre-timed Webster method and the myopic PressLight RL method across all demand levels, achieving substantial reductions in Total Time Spent (TTS) and Total Queue Time. For instance, in the heavily saturated Network 1x3, the 1-hop method reduced TTS by approximately 10% compared to PressLight.
- Improved Congestion Management: Figure 7 clearly shows that the farsighted (1-hop) agent leads to lower peak queue lengths and more balanced eastbound/southbound queues compared to the myopic (0-hop) agent.
- Toronto Testbed Success: In the realistic Toronto scenario, the 2-hop and 3-hop RL models significantly outperformed the City Plan baseline, reducing TTS by 6% and 19% respectively. Even the 1-hop model showed considerable improvement over the City Plan.
- Robustness: The method demonstrated robustness across different demand levels and network complexities.
Limitations and Future Work
- Limitations:
- The paper notes that while the multi-hop pressure metric is more compact than neural network-based communication, it still adds complexity compared to myopic methods.
- The study primarily focuses on potential-based rewards, acknowledging that pressure-based rewards can lead to higher variability and instability.
- Future Work:
- The authors suggest incorporating dynamic turning ratio estimation.
- Expanding the scope to more complex networks is proposed.
- Applying multi-hop pressure to other traffic control problems like congestion pricing and dynamic perimeter identification is also suggested.
- Further refinement of the approach by exploring higher hop counts (e.g., 2-hop and 3-hop) in more complex scenarios is also indicated.
Summary updated