Abstract
Queuing network control is a critical component of modern infrastructure, from semiconductor manufacturing to healthcare systems. However, applying reinforcement learning (RL) to these environments has been historically difficult due to the lack of standardized benchmarks and the inherent instability of RL in continuous-time stochastic systems. We introduce QGym, an open-source simulation framework designed to bridge this gap. By implementing an event-driven architecture and introducing a "work-conserving" inductive bias (PPO-WC), we demonstrate that RL can outperform traditional control policies in 77% of tested scenarios, particularly in high-variance, real-world environments.
Introduction
Managing congestion in job-processing systems is a delicate balancing act. In environments like hospital wards or telecommunication switches, jobs arrive at random intervals and must be efficiently routed to available servers. If a controller makes suboptimal decisions, queues do not merely grow—they can "explode," leading to system instability and infinite delays.
Historically, this field has relied on classical policies such as the $c\mu$-rule or MaxWeight. While these methods provide mathematical guarantees, they often rely on oversimplified Markovian assumptions (e.g., exponential distributions) that do not hold in the real world. While Reinforcement Learning (RL) offers a data-driven alternative, it has struggled with the long planning horizons and high randomness of queuing systems. QGym provides the first scalable, standardized environment to benchmark RL against these traditional methods across diverse, realistic scenarios.
The QGym Framework
QGym is built on the OpenAI Gym interface, making it immediately accessible to the RL research community. It distinguishes itself from standard RL environments through several key technical features:
- Event-Driven Simulation: Rather than updating at fixed time steps, QGym utilizes an event-driven engine that jumps directly to the next meaningful event (an arrival or service completion). This is computationally efficient and provides the precision required for continuous-time systems.
- Job-Level Tracking: The simulator tracks individual jobs throughout their lifecycle, allowing for the modeling of complex behaviors such as preemption (interrupting a job) and server pools (multiple servers servicing a single queue).
- Work-Conserving Inductive Bias: A primary contribution of this work is the identification of why vanilla RL fails in queuing. The authors introduce PPO-WC (Work-Conserving PPO), which applies a mask to the agent's action space. This prevents the agent from assigning a server to an empty queue, ensuring it does not waste resources—a fundamental principle of queuing theory that vanilla agents often fail to learn from scratch.
Experimental Setup and Datasets
QGym includes a suite of 20 unique problem instances to test the limits of various controllers:
- Classical Networks: Standard theoretical models like Criss-cross, Tandem, and Re-entrant networks common in manufacturing.
- Five-by-Five Network: A call-center model featuring time-varying arrival rates.
- Real-World Hospital System: A high-fidelity simulation calibrated to a real hospital, featuring 8 specialties, 11 wards, and 497 servers (beds).
- Input-Queue Switch: A model for packet routing in high-speed crossbar switches.
Key Insights and Results
The authors compared PPO-WC against four traditional baselines: $c\mu$, MaxWeight, Max Pressure, and Fluid Policy.
Stability and Performance
While vanilla PPO led to exploding queues in every single setting, the PPO-WC agent remained stable and outperformed the best traditional policy in 77% of all instances. In re-entrant manufacturing models, PPO-WC consistently discovered scheduling patterns superior to the $c\mu$-rule, which is often considered the gold standard for such systems.
Robustness to Noise
Traditional policies are typically optimized for exponential distributions. When the researchers introduced "hyper-exponential" noise (representing higher variance), the performance gap widened. In these high-noise scenarios, RL achieved up to a 21% reduction in costs compared to classical methods, suggesting that RL is significantly better at adapting to the "messiness" of real-world data.
The Scaling Challenge
Despite these successes, a "scaling wall" remains. In the massive hospital system model—the most complex environment in the suite—the RL agent was unable to beat the best classical heuristics even after 5 million training steps. This highlights a persistent sample inefficiency problem when applying RL to industrial-scale networks.
Conclusion and Future Work
QGym demonstrates that with the right inductive biases, RL can significantly improve upon decades of classical queuing theory, particularly in non-idealized environments. However, the difficulty of scaling to massive systems like hospitals suggests that the future of queuing control may lie in "hybrid" controllers. These would combine the mathematical stability of classical rules with the flexible, data-driven learning capabilities of RL to manage the world's most complex infrastructures.
Summary updated