Multi-hop Upstream Preemptive Traffic Signal Control with Deep Reinforcement Learning

Xiaocan Li, Xiaoyu Wang, Ilia Smirnov, Scott Sanner, Baher Abdulhai

Introduction

Traffic signal control (TSC) is a critical component of urban mobility. Traditional methods often rely on "traffic pressure"—the difference in vehicle density between an intersection's incoming and outgoing links. While effective for local optimization, these metrics are inherently myopic, focusing only on immediate neighbors. This lack of spatial awareness often leads to "stop-and-go" waves, as controllers fail to account for massive traffic jams brewing just a few blocks away.

To address this, we introduce multi-hop upstream pressure, a novel metric grounded in Markov chain theory. By integrating this farsighted metric into a Deep Reinforcement Learning (DRL) framework, traffic agents can "see" congestion several intersections away, allowing for preemptive queue clearing and improved network coordination.

The Multi-hop Pressure Framework

The core innovation lies in modeling the traffic network as a graph where vehicle movements are transitions in an absorbing Markov chain.

Markov chain representation of traffic flow with a supersink node.

As shown in the figure above, the network includes a "supersink" node ($\Omega$) representing all exits, ensuring the transition matrix remains mathematically sound. By utilizing the powers of the transition matrix ($P^h$), we derive a vectorized formula to calculate the influence of traffic $h$ hops away. This approach allows the system to calculate pressure for the entire network simultaneously and efficiently.

This multi-hop pressure is then fed into a Proximal Policy Optimization (PPO) agent. Unlike standard pressure-based rewards that might encourage agents to "hold" vehicles upstream, our potential-based reward incentivizes the downstream movement of vehicles, effectively clearing not just local queues but the "potential" traffic heading toward the intersection.

Experimental Evaluation

The proposed approach was evaluated using the Aimsun traffic simulator across both synthetic arterial networks and a realistic Toronto testbed.

Simulation of an arterial traffic network in Aimsun.

The Toronto testbed modeled a neighborhood (Sheppard Avenue and Highway 404) with 12 signalized intersections using real-world demand profiles and calibrated turning counts. Synthetic tests further explored performance under varying saturation levels, from undersaturated to heavily congested scenarios.

Key Results and Insights

The multi-hop RL model demonstrated significant performance gains, particularly in high-congestion scenarios:

  • Toronto Testbed: The 3-hop model achieved a 19% improvement in Total Time Spent (TTS) compared to the actual Toronto "City Plan" and vastly outperformed myopic RL baselines.
  • Saturation Resilience: In heavily saturated synthetic tests, the model reduced Total Virtual Queue Time by over 50%.
  • Preemptive Clearing: Farsighted agents were observed assigning more green time to heavy-flow directions before the upstream wave arrived, preventing the formation of massive virtual queues.

By using a compact, mathematically derived metric instead of complex communication protocols (like Graph Neural Networks), the system achieves spatial awareness without the training instability often associated with multi-agent communication.

Conclusion and Future Work

Multi-hop upstream pressure provides a robust, farsighted alternative to traditional TSC metrics. While the current model relies on static turning ratios and requires reliable vehicle count data, it paves the way for more proactive urban traffic management. Future research will explore dynamic turning ratios and the integration of this framework into larger, more complex grid networks.

Summary updated

BibTeX
@article{li2025multi,
  author = {Xiaocan Li and Xiaoyu Wang and Ilia Smirnov and Scott Sanner and Baher Abdulhai},
  title = {{Multi-hop Upstream Preemptive Traffic Signal Control with Deep Reinforcement Learning}},
  journal = {IEEE Open Journal of Intelligent Transportation Systems},
  year = {2025},
  doi = {10.1109/ojits.2025.3562757},
}
Figure