Abstract
Urban traffic signal control remains a significant challenge due to the diversity of intersection topologies and the difficulty of coordinating massive networks. This paper introduces OAM (Option-Action Reinforcement Learning), a framework that achieves universal traffic control by disentangling complex traffic phases into simple, lane-level "options." By leveraging a physics-informed state representation and a mathematically grounded reward function, OAM demonstrates superior performance and scalability across networks of up to 2,048 intersections.
Introduction
Reinforcement learning (RL) has emerged as a powerful tool for optimizing traffic signals. However, existing models often struggle with three critical issues:
- Generalization: Most models are tailored to specific intersection layouts (e.g., 4-way crosses) and fail when applied to irregular topologies like 3-way or 5-way junctions.
- Reward Design: Heuristic rewards like queue length do not mathematically guarantee the minimization of total travel time.
- Scalability: Coordinating thousands of signals is computationally intensive, and decentralized approaches often lack a theoretical basis for effective cooperation.
OAM addresses these challenges by treating intersections as collections of lanes rather than fixed topologies, enabling a single model to control any junction layout.
Methodology
The OAM framework introduces three primary innovations to create a robust, universal controller.
Lane-Cell State Representation
Instead of treating a lane as a monolithic block, OAM adopts the Cell Transmission Model (CTM). Lanes are divided into "cells" based on speed limits, allowing the model to capture the physics of traffic flow. This representation enables the agent to understand not just the volume of traffic, but its exact spatial distribution and propagation speed.
Option-Action Disentanglement
This is the core mechanism for universality. Rather than selecting a specific "Phase" (e.g., North-South Green), the model evaluates three Lane Options for every possible movement: Keep (maintain green), Switch (change to green), or Stop (remain red).
As shown in the architecture above, the model computes values for these options at the lane level. Because any intersection is fundamentally a set of lanes, the same neural network can be applied to any topology by aggregating these lane-level values to determine the optimal overall phase.
Regularized Delay Reward and Coordination
OAM utilizes a "regularized" reward function combining negative delay with an "outflow" term (the number of vehicles exiting a lane). The authors provide mathematical proof that this reward is equivalent to minimizing average travel time while providing a stronger learning signal. For multi-intersection coordination, OAM employs potential game theory to derive a decentralized Q-function, focusing coordination efforts on scenarios involving queue spillback.
Evaluation and Results
The framework was rigorously tested using the CityFlow simulator across various real-world datasets, including Jinan, Hangzhou, and Manhattan.
A significant highlight of the evaluation was a city-scale challenge in Nanchang, involving 2,048 intersections with diverse topologies and road lengths.
Key Findings
- Superior Performance: OAM consistently achieved the lowest average travel times. In the Nanchang city-level test, it outperformed state-of-the-art models like MPLight and AttendLight in 8 out of 9 scenarios.
- Efficiency and Robustness: Due to its universal structure, OAM can train on data from all intersections simultaneously, leading to faster convergence. It also maintained performance better than competitors as traffic volumes increased from 2,000 to 6,000 vehicles.
Conclusion and Future Work
OAM represents a significant step toward "topology-invariant" traffic management. By modeling at the lane level, it allows for seamless deployment across different city layouts without retraining. The inclusion of "outflow" in the reward function effectively solves the credit assignment problem in RL-based traffic control.
Future research will focus on relaxing the assumption that vehicles cannot cross two intersections within a 10-second interval. This will allow the framework to handle extremely dense "micro-grids" where intersections are very close together, further enhancing its applicability to complex urban environments.
Summary updated