Introduction
Bus bunching is a pervasive issue in urban transit systems where buses on the same route lose their scheduled spacing, eventually arriving at stops in clusters. This phenomenon is inherently unstable: a slight delay increases the number of passengers at the next stop, further slowing the bus, while the following bus speeds up due to fewer pickups. To combat this, transit agencies use "bus holding" strategies, intentionally delaying buses at stops to restore even headways.
While Multi-Agent Reinforcement Learning (MARL) offers a promising path for optimizing these holding times, real-world bus systems are asynchronous. Unlike standard MARL environments where agents act in synchronized turns, a bus only makes a decision upon arriving at a stop. This temporal misalignment makes it difficult for traditional algorithms to assign credit—determining how a specific action by one bus at one time affects the system's overall performance later.
The Challenge of Asynchronous Control
In a typical bus route, agents (buses) operate independently and trigger control events at different times. Standard MARL critics struggle to evaluate these staggered actions because the state of the environment is constantly shifting between decision points.
As shown in the figure above, the control logic must account for the fact that feedback is received after a delay, and the impact of a holding action ripples through both upstream and downstream agents. To address this, researchers developed the Credit Assignment framework for Asynchronous Control (CAAC).
The CAAC Framework
The core innovation of CAAC is its ability to handle asynchronous interactions through a specialized architecture. It moves away from fixed time-steps and instead treats the system as a dynamic Event Graph (EG), where nodes represent arrival events and edges represent spatiotemporal dependencies between buses.
Dual-Critic Architecture
To effectively assign credit in this asynchronous environment, CAAC employs a dual-critic structure:
- Ego Critic: This component evaluates the bus's own local policy and immediate state, focusing on the direct consequences of its holding actions.
- Event Critic: Utilizing Graph Attention Networks (GAT), the Event Critic analyzes the Event Graph to calculate the "marginal contribution" of other buses. It identifies how the actions of nearby agents (within a specific time window) influence the current bus's performance.
By separating "what I did" from "how others affected me," the model achieves more stable training and a clearer understanding of multi-agent coordination. Furthermore, because the GAT-based critic learns general relationships rather than specific schedules, the framework supports inductive learning, allowing a model trained on one route to be deployed on another without retraining (zero-shot transferability).
Experimental Evaluation
The framework was validated using real-world smart card data from four major trunk routes (R1–R4) in a large city. A custom simulator, calibrated to match real-world boarding patterns and travel uncertainties, served as the testing ground. CAAC was compared against several baselines, including "No Control," traditional headway-based methods, and standard MARL algorithms like MADDPG.
Key Results
- Service Reliability: CAAC achieved the lowest Average Waiting Time (AWT) and Average Occupancy Dispersion (AOD) across all tested routes, indicating more reliable service and more evenly distributed passenger loads.
- Robustness and Transferability: A model trained solely on Route R1 outperformed all baselines when tested on Routes R2, R3, and R4. For instance, on Route R2, CAAC significantly mitigated the increase in waiting time compared to the "No Control" scenario, demonstrating its ability to generalize across different route geometries and bus counts.
Conclusion
CAAC represents a significant step forward in applying MARL to real-world, asynchronous infrastructure. By leveraging graph-based credit assignment, it effectively manages the complex, staggered interactions inherent in transit systems. Future work may involve integrating traffic signal priority and exploring more complex passenger demand patterns to further enhance urban mobility.
Summary updated