Multi-Agent Reinforcement Learning for Traffic Signal Control through Universal Communication Method

Qize Jiang, Minhao Qin, Shengmin Shi, Weiwei Sun, Baihua ZhengIJCAI 2022 · DOI

TL;DR

This paper introduces "UniComm," a novel communication method for multi-agent traffic signal control that focuses on sharing predictive impacts rather than raw observations. Coupled with a new lightweight network architecture, "UniLight," this approach significantly improves traffic flow efficiency and robustness across various scenarios.

What Problem Does This Paper Address?

Traffic congestion is a persistent urban challenge. While traffic signals help manage flow, traditional timer-based systems are inefficient in dynamic traffic conditions. Multi-agent reinforcement learning (MARL) offers a promising solution for coordinating multiple intersections, but it faces challenges like nonstationarity and the curse of dimensionality. A key hurdle in MARL for traffic control is effective communication between agents (traffic signals). Existing methods often share raw observations or learned hidden states, which can be inefficient, lead to slow convergence, and don't guarantee that the most important information is shared. The authors identify a critical gap: the lack of a theoretically grounded and content-aware communication mechanism that prioritizes information relevant to neighboring intersections' performance.

Methods

The paper proposes two main innovations:

  • UniComm (Universal Communication Method): Instead of sharing raw sensor data or abstract hidden states, UniComm focuses on predicting the impact of an intersection's traffic movements on its neighbors. Specifically, an agent predicts how its current traffic observations and actions will affect the approaching vehicle counts in its neighbors' incoming roads. This prediction is then shared. The core idea is that only information directly relevant to a neighbor's future state is valuable, making communication more efficient and targeted. This method is designed to be "universal," meaning it can be integrated with existing MARL algorithms.
  • UniLight: To effectively leverage UniComm, the authors introduce UniLight, a concise network architecture. UniLight processes incoming traffic data and UniComm's predictions to estimate the Q-values for different traffic signal phases. It prioritizes traffic movements that are permitted by a given phase and assigns them higher importance, reflecting their direct impact on traffic flow. This selective weighting helps the agent make more informed decisions.

The underlying MARL framework uses Multi-Agent Deep Q Learning with double Q-learning and dueling networks.

Datasets

The experiments were conducted using real-world traffic datasets from four cities:

  • Hangzhou (HZ), Jinan (JN), and Shanghai (SH) from China.
  • New York (NY) from the USA.

The SH dataset was specifically constructed to simulate a more realistic urban environment. Datasets from HZ, JN, and NY are publicly available and commonly used in related research.

Key Insights and Ideas

  • Content over Quantity: The most significant insight is that the content and relevance of shared information are far more critical than the sheer amount of data exchanged. UniComm's success stems from its focus on predictive impact, not just raw data.
  • Predictive Communication: Sharing predictions of future traffic states (approaching vehicles) is more effective than sharing current observations or abstract hidden states. This allows neighboring agents to proactively adjust their strategies.
  • Universality of UniComm: The proposed UniComm method is designed to be plug-and-play, improving the performance of various existing MARL algorithms without requiring them to be fundamentally redesigned.
  • Efficiency of UniLight: The UniLight architecture demonstrates that a focused, lightweight network can effectively utilize the enhanced communication provided by UniComm, leading to superior performance.
  • Robustness in Complex Environments: The proposed methods show particular strength in more complex traffic scenarios (like SH1/SH2), where simpler or less communication-aware methods struggle.

Key Results

  • UniComm Boosts Performance: When applied to six representative RL-based MARL methods, UniComm consistently improved their performance, reducing average travel time.
  • UniLight Outperforms: UniLight, using UniComm, achieved the best performance across almost all datasets, demonstrating significant advantages in complex environments. It improved average performance by 8.2% on three public datasets and 35.6% on more complicated ones (SH1/SH2).
  • UniComm vs. Hidden States: UniLight with UniComm outperformed a version of UniLight that communicated 32-dimensional hidden states, even though UniComm shared less information. This reinforces the "content over quantity" insight.
  • Phase Prediction Improvement: Using actions stored in the replay buffer as a target for phase prediction (instead of the current network's action) led to more stable and accurate phase predictions.

Limitations and Future Work

  • One-Step Prediction: The paper currently focuses on one-time-step predictions for communication. The authors acknowledge that predicting further into the future is challenging due to the nature of off-policy learning and the potential invalidation of assumptions over longer horizons. Future work could explore multi-step prediction strategies.
  • Assumptions: The method relies on two assumptions: no spillback and that vehicles do not traverse an entire road within one action interval. While these simplify the problem and are often realistic, their violation in extreme cases might affect performance.
  • Dataset Limitations: While real-world datasets were used, the paper notes that some datasets are simulated or have limitations (e.g., short simulation times, small road networks). The SH dataset was created to address some of these, but further validation on even larger and more diverse real-world urban traffic networks would be beneficial.

The authors suggest that their UniComm and UniLight framework provides a strong foundation for future research in decentralized multi-agent traffic signal control.

Summary updated

BibTeX
@inproceedings{jiang2022multi,
  author = {Qize Jiang and Minhao Qin and Shengmin Shi and Weiwei Sun and Baihua Zheng},
  title = {{Multi-Agent Reinforcement Learning for Traffic Signal Control through Universal Communication Method}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2022},
  doi = {10.24963/ijcai.2022/535},
}
Figure