Abstract
Traffic light control is a complex optimization task characterized by non-stationary environments where traffic patterns shift over time. This paper evaluates Reinforcement Learning with Context Detection (RL-CD) in a microscopic simulation environment. By maintaining a library of specialized models and detecting context shifts through prediction error, RL-CD aims to overcome the limitations of standard RL in adapting to new traffic regimes. Our results, obtained using the ITSUMO simulator with Nagel-Schreckenberg driver behavior, demonstrate that RL-CD significantly outperforms traditional methods in noisy, realistic scenarios, although high levels of driver randomness can lead to state aliasing.
1. Introduction
Traffic light control is notoriously difficult because traffic is inherently non-stationary; patterns change throughout the day, such as during morning rush hours versus late-night lulls. Standard Reinforcement Learning (RL) algorithms, like Q-learning, typically assume a stationary environment. When traffic patterns shift, these agents must "unlearn" their previous policies, leading to performance degradation during transitions. Furthermore, many existing studies rely on macroscopic models that treat traffic as a fluid. This research addresses these challenges using microscopic simulation, which models individual vehicle interactions and incorporates "braking noise"—the random, sub-optimal decelerations of real drivers—adding a layer of unpredictable noise that complicates the learning process.
2. Methodology: RL-CD
The proposed approach utilizes Reinforcement Learning with Context Detection (RL-CD). Rather than learning a single universal policy, RL-CD maintains a library of partial models tailored to specific traffic contexts.
- Context Detection: The system monitors the prediction error of the active model. If the error exceeds a predefined "plasticity threshold" ($\lambda$), a change in the environment is assumed.
- Model Switching and Creation: Upon detecting a shift, the agent searches its library for a model that better fits the new observations. If no suitable model is found, a new one is initialized.
- Prioritized Sweeping (PS): To enhance efficiency, the agent employs PS, a model-based RL technique that updates knowledge more effectively than standard model-free Q-learning.
- Microscopic Simulation: The Nagel-Schreckenberg model is used to simulate realistic driver behavior. A deceleration probability parameter ($p$) allows for the tuning of environmental noise.
3. Experimental Setup
The study utilizes a synthetic benchmark generated via the ITSUMO microscopic traffic simulator. The network topology consists of a 3x3 Manhattan-style grid with 9 intersections and 24 one-way links.
The experiments involve three distinct traffic contexts:
- Low: Low volume across all directions.
- Vertical: High volume originating from the North.
- Horizontal: High volume originating from the East.
To evaluate robustness, four noise levels were tested by varying the deceleration probability ($p = 0, 0.1, 0.2, 0.3$).
4. Results and Discussion
The core insight of this work is that traffic management can be decomposed into a series of simpler, stationary problems. By partitioning knowledge into contexts, the agent retains specialized policies for different times of day.
Key findings include:
- Performance in Noise: In moderate noise ($p=0.1$), RL-CD and Prioritized Sweeping significantly outperform both greedy strategies and standard Q-learning. RL-CD successfully identifies context shifts and adapts its policy rapidly.
- The Impact of High Noise: As driver randomness increases ($p \ge 0.2$), performance across all learning methods degrades. At $p=0.3$, "state aliasing" occurs; sensors (inductive loops) report lanes as "full" regardless of the actual pattern, making it impossible for the agent to distinguish between contexts.
- Q-Learning Limitations: Standard Q-learning performs poorly due to its model-free nature, spending excessive time exploring suboptimal actions during the learning phase, which exacerbates congestion.
5. Conclusion and Future Work
While RL-CD shows promise, its effectiveness is limited by coarse state discretization. In saturated or high-noise scenarios, the binary or ternary state descriptions (e.g., "empty" vs "full") prevent the context detection mechanism from functioning correctly.
Future research should focus on:
- Finer-grained state descriptions to improve context differentiation in heavy traffic.
- Goal-driven driver agents that can learn to adapt to traffic signals, creating a co-evolutionary environment.
- Decentralized Coordination to manage network-wide traffic flow through inter-agent communication.
Summary updated