Abstract
Dynamic scheduling is a critical task in various domains, from wireless networks to resource management. However, conventional Reinforcement Learning (RL) policies are often rigid, tied to a fixed number of items and specific system dynamics. This paper introduces the Descriptive Policy framework, which shifts the focus from item identities to their underlying conditions. By representing states and actions in a system-agnostic "condition space," the proposed method enables agents to learn universal scheduling principles that generalize across different systems without retraining.
Introduction: The Challenge of Dynamic Scheduling
Dynamic scheduling involves selecting an item from a set at each timestep to maximize long-term utility. While Markov Decision Processes (MDPs) provide a robust mathematical foundation for these problems, standard RL solutions suffer from a lack of flexibility. A policy trained for a network with five users typically cannot handle ten users without significant architectural changes or retraining. This "system-specific" nature limits the deployment of RL in real-world environments where system dynamics and the number of participants are fluid.
The Descriptive Policy Framework
The core innovation of this work is the decoupling of scheduling logic from specific item indices. This is achieved through a two-fold transformation:
1. Descriptive State ($\bar{s}$)
Instead of processing a raw list of features for $N$ items, the system maps the state $s$ to a descriptive state $\bar{s}$. This state is a multi-dimensional matrix where each dimension corresponds to a feature (e.g., price, quantity) partitioned into discrete bins. A cell in this matrix contains a binary indicator: '1' if at least one item falls into that specific condition bin, and '0' otherwise.
2. Descriptive Action ($\bar{a}$)
Similarly, the policy does not select an item index. It selects a "condition" (a cell in the descriptive matrix) to serve. A translation function $d_a$ then maps this descriptive action back to a physical item in the system that meets the chosen criteria.
This abstraction ensures that the input and output dimensions of the policy remain constant, regardless of the number of items ($N$) in the system.
Experimental Evaluation
The researchers tested the Descriptive Policy in two distinct environments:
- Synthetic Sales Scenario: A scheduler decides which item to sell based on price and quantity. Testing involved systems with 2, 6, and 10 items.
- Wireless Network Scenario: A base station schedules users with varying channel conditions and data rate requirements. This scenario included complex factors like pathloss and shadowing, with user counts ranging from 4 to 20.
Key Results and Discussion
The Descriptive Policy demonstrated several advantages over conventional index-based policies:
- Zero-Shot Generalization: Policies trained on smaller systems (e.g., $N=2, 6$) performed optimally on larger systems ($N=10$) immediately, demonstrating that the agent learned general "scheduling principles" rather than system-specific patterns.
- Performance Stability: Unlike conventional policies that require extensive re-learning when system parameters change, the Descriptive Policy maintains high average rewards across transitions between different system dynamics.
- Explainability: The use of a condition matrix allows for direct visualization of the agent's decision-making process. By examining Q-values across the descriptive state space, researchers can identify which feature combinations the policy prioritizes.
Conclusion and Future Directions
The Descriptive Policy framework offers a scalable and system-agnostic approach to dynamic scheduling. By focusing on conditions rather than identities, it achieves remarkable robustness and generalization. Future work will address the sensitivity of the model to bin discretization and explore automated methods for feature partitioning to further enhance performance in highly heterogeneous environments.
Summary updated