TL;DR
HALight is a novel multi-agent reinforcement learning (MARL) framework designed for traffic signal control that breaks away from the "idealized" assumptions of previous models. By supporting heterogeneous intersection modeling and a lightweight neighbor-communication mechanism, it maintains high traffic efficiency even when intersections have diverse layouts and sensors frequently fail to provide complete data.
What Problem Does This Paper Address?
Most existing reinforcement learning (RL) methods for traffic signal control (TSC) operate under two major simplifications that rarely hold true in the real world:
- Homogeneity: They assume every intersection is identical in terms of lane configuration and signal phases. In reality, cities are a mix of T-junctions, standard crosses, and complex multi-lane intersections.
- Full Observability: They assume sensors (like cameras or induction loops) are perfect. In practice, sensor failures, limited coverage, or "blind spots" in electronic toll collection (ETC) sensing lobes lead to missing lane-level data.
When a standard RL agent trained on "average" intersection data meets a unique T-junction, or when its sensors go dark, its performance degrades rapidly, leading to traffic instability and poor coordination.
Methods
The authors propose HALight, a framework that tackles these issues through three core architectural choices:
- Heterogeneous Actor Modeling: Instead of a one-size-fits-all policy, HALight allows for intersection-specific observation and action spaces. This means an agent at a T-junction isn't trying to use a policy designed for a 4-way cross.
- Communication-Enhanced Coordination: To compensate for missing local data, HALight incorporates a lightweight, one-hop communication module. Using a dot-product attention mechanism, agents selectively aggregate "hypotheses" from their neighbors. If an agent's own sensors are missing data for an incoming lane, the information from the upstream neighbor helps fill in the blanks.
- Sequential Policy Updates: Drawing from Heterogeneous-Agent Proximal Policy Optimization (HAPPO), HALight uses a sequential update scheme. Rather than all agents updating simultaneously—which can be unstable in a multi-agent environment—they update one by one, accounting for the changes made by previously updated neighbors.
- Temporal Modeling: Each agent uses a Gated Recurrent Unit (GRU) to maintain a memory of past traffic states, helping it infer current conditions even during intermittent sensor outages.
Datasets
The model was evaluated using the CityFlow simulation platform on two real-world road networks from Hangzhou, China:
- Xiasha District: A 3×3 grid featuring 7 crossing intersections and 2 T-junctions with varying configurations.
- Gudang District: A 4×4 grid of 16 crossing intersections with diverse structural layouts.
The researchers tested these networks under both "moderate" and "high" traffic demands, with lane-level observation missing rates ranging from 0% to 40%.
Key Insights and Ideas
- Spatial-Temporal Compensation: The combination of GRUs (temporal) and neighbor communication (spatial) acts as a powerful redundancy system. When local "eyes" fail, the agent uses its "memory" and its "neighbors' advice" to make a decision.
- Avoiding the "Average" Trap: Homogeneous models tend to learn a policy that is a mediocre compromise for all intersections. By embracing heterogeneity, HALight allows agents to specialize, which is particularly effective in structurally diverse networks like Xiasha.
- Lightweight is Better: Unlike complex Graph Attention Network (GAT) approaches that require global information, HALight’s one-hop communication is computationally efficient and scales linearly with the number of intersections, making it viable for large-scale city deployment.
Key Results
HALight consistently outperformed both homogeneous (e.g., PressLight, DynSTGAT) and heterogeneous (e.g., HAPPO) baselines:
- Robustness to Data Loss: At a 40% missing observation rate, many baseline models failed to converge or showed massive performance spikes. HALight remained stable, maintaining significantly lower Average Travel Time (ATT) and Average Queue Length (AQL).
- Efficiency: In the Gudang high-demand scenario with 40% data loss, HALight achieved an ATT of ~314 seconds, whereas the best-performing baseline (Het-MAPPO) lagged behind at ~317 seconds, and others failed entirely.
- Communication Resilience: The model proved robust to communication imperfections, showing only minor performance drops even with 50% packet loss or significant transmission delays.
Limitations and Future Work
While HALight is a significant step toward realistic TSC, the authors note a few areas for improvement:
- More Complex Traffic: The current study uses relatively standard phase plans. Future work could explore richer, more adaptive phase designs.
- Long-Range Interactions: The current model only looks at one-hop neighbors. In very dense or highly congested networks, multi-hop communication might be necessary to manage traffic waves that propagate across several blocks.
- Adaptive Communication: Exploring ways for agents to decide when and what to communicate more selectively could further optimize bandwidth usage in real-world IoT deployments.
Summary updated