TL;DR
This paper introduces a novel deep reinforcement learning (RL) approach to automatically design effective and generalizable scheduling functions (SFs) for Software-Defined Networking (SDN) with distributed controllers. By representing the SF as a neural network that assigns priorities to controllers and using a unique probability selection scheme, the method efficiently dispatches requests to optimize network performance across diverse network configurations.
What Problem Does This Paper Address?
Software-Defined Networking (SDN) is increasingly adopting distributed controller architectures to enhance scalability and performance. A critical component in these systems is the scheduling function (SF), which determines how requests from network switches are dispatched to available controllers. The challenge lies in designing SFs that are both highly efficient (minimizing request response time) and generalizable, meaning they can perform well across various network settings with different numbers and capacities of controllers. Existing approaches often rely on manually designed heuristics or require extensive, time-consuming retraining for each new network configuration, failing to meet these crucial requirements.
Methods
The authors propose a two-part RL-based approach:
- Dispatching System Design: Instead of directly mapping network states to controller choices, the SF is designed as a neural network (NN) that outputs a "priority" for each controller. This priority is then used by a novel probability selection scheme to determine which controller receives a request. This scheme balances exploration (trying different controllers) and exploitation (using the best-performing ones) by assigning non-zero probabilities only to controllers with high priorities, controlled by a hyperparameter
m. This approach allows the SF to adapt to networks with varying numbers of controllers. - Training System: The SF is trained using Proximal Policy Optimization (PPO), a state-of-the-art actor-critic RL algorithm. The training process involves simulating various network settings (defined by controller capacities, arrival rates, and communication delays) and using PPO to optimize the SF's parameters. A key innovation is a new technique for calculating the gradient of the SF, which is crucial for efficient training with PPO. The training is designed to be adaptive, improving the SF's performance across different network configurations.
Datasets
Not discussed in the paper. The authors use simulated network environments for training and testing.
Key Insights and Ideas
- Priority-based SF with Probability Selection: Representing the SF as a controller priority predictor, rather than a direct dispatcher, combined with a probability selection mechanism (like a modified softmax or Euclidean projection), is key to achieving generalization. This allows the system to dynamically weigh controller suitability without being rigidly tied to a fixed number of outputs.
- Balancing Exploration and Exploitation: The proposed probability selection scheme, particularly the use of the
mhyperparameter, offers a principled way to control the trade-off between exploring new controller options and exploiting known good ones, which is vital for effective RL training. - Adaptive Training for Generalization: Training the SF across a diverse set of network settings (Env 1-3) is crucial for developing a SF that can generalize well to unseen network configurations (Env 4-7). This adaptive training approach is a significant departure from methods that require re-engineering for each new scenario.
- NNs for SFs are Efficient: The paper argues that small, feed-forward NNs can meet the real-time efficiency requirement (R1) for SFs, especially when supported by optimized software and hardware.
Key Results
- The RL-based SF can be rapidly trained to achieve optimal performance, as demonstrated by the quick convergence of cumulative reward and decrease in average response time in initial training runs (Fig. 3).
- The trained SF generalizes well to network settings it was not explicitly trained on. For example, in a South American network (Env 4), the proposed SF achieved a response time of 60ms, a 40% improvement over random and weighted round-robin heuristics (which achieved ~105ms).
- The proposed SF consistently outperforms traditional heuristics like Random and Weighted Round-Robin across various network topologies and traffic loads (Fig. 4). This is attributed to the SF's ability to consider both controller capacity and communication delay, whereas heuristics often neglect one or the other.
- In specific scenarios (Fig. 5), the proposed SF intelligently distributes requests to controllers with lower communication delays, leading to significantly lower response times (30ms vs. 96ms and 108ms for heuristics).
Limitations and Future Work
- Hyperparameter Tuning: The paper mentions that
mis a hyperparameter for the probability selection scheme, implying that its optimal value might need tuning. - Simulation-based: The results are based on simulations. Real-world deployment and validation would be the next logical step.
- Future Work: The authors suggest that their approach can be extended to other RL algorithms and that further research could explore more complex network dynamics and controller interactions. The paper also notes that while PPO is used, other RL algorithms could be investigated.
Summary updated