Online Service Provisioning in NFV-Enabled Networks Using Deep Reinforcement Learning

Ali Nouruzi, Abolfazl Zakeri, Mohammad Reza Javan, Nader Mokari, Rasheed Hussain, S. M. Ahsan KazmiIEEE Trans. Netw. Serv. Manag 2022 · DOI

TL;DR

This paper tackles the challenge of efficiently provisioning online service requests in Network Function Virtualization (NFV) enabled networks, considering dynamic resource availability and probabilistic service lifetimes. They propose a Deep Q-Network (DQN) based approach, DQN-AR, that intelligently allocates network resources for function placement and dynamic routing, leading to improved service acceptance and reduced resource utilization costs.

What Problem Does This Paper Address?

Modern networks are increasingly supporting applications with diverse Quality of Service (QoS) requirements, such as low latency and high bandwidth. Network Function Virtualization (NFV) offers a flexible and cost-effective way to deploy these services by virtualizing network functions (VNFs) and chaining them together to form Network Services (NSs). However, efficiently managing and allocating resources in these dynamic NFV environments is a significant challenge. Specifically, this paper addresses the problem of online service provisioning, where service requests arrive dynamically and have a probabilistic "lifetime" (duration). The core challenge is to fulfill these requests with their QoS requirements while minimizing the cost of network resource utilization, especially when resources are limited and service demands are stochastic. Existing approaches often struggle to handle the dynamic nature of service lifetimes and online arrivals in a cost-effective and intelligent manner.

Methods

The authors propose a Deep Q-Network for Adaptive Resource allocation (DQN-AR) algorithm. This approach leverages Deep Reinforcement Learning (DRL) to make intelligent decisions about function placement and dynamic routing.

Here's a breakdown of the key methodological components:

  • Problem Formulation: The problem is framed as an optimization problem aiming to minimize the cost of network resource utilization while satisfying QoS constraints.
  • DRL Agent: The SDN controller acts as the DRL agent. It observes the network state and selects actions to provision services.
  • State Space: The network state is defined by the available resources, including processing capacity of Virtual Machines (VMs) on nodes and bandwidth of links. This state is normalized and fed into the DQN.
  • Action Space: The action space is complex, encompassing decisions for both function placement (assigning a VNF to a specific VM on a node) and dynamic routing (selecting the next node in the service function chain). The agent can also choose to use a node as a simple forwarding device (switch).
  • DQN Algorithm: A Deep Q-Network (DQN) is employed to estimate the Q-values, which represent the expected future rewards for taking a particular action in a given state. This is crucial for handling the large state-action space inherent in NFV environments.
  • Adaptive Resource Allocation: The DQN-AR algorithm dynamically allocates resources. It considers the service's specific requirements (SFC, QoS) and the current network conditions.
  • Online Provisioning: The algorithm is designed for online provisioning, meaning it makes decisions as requests arrive and can handle new requests while existing ones are being served.
  • Reward Function: A piecewise reward function is defined, which incentivizes the agent to satisfy constraints (like QoS and resource availability) and minimize costs.

Datasets

The paper does not explicitly mention using pre-existing datasets for training. Instead, it relies on simulations to evaluate the proposed DQN-AR algorithm. The simulation setup is described in detail, including parameters for service specifications (duration, data rate, tolerable time), network resources (VM capacity, link capacity), and network configuration (number of nodes, VMs per node, propagation delay). The arrival of service requests is modeled using a Uniform random process, and service lifetimes are modeled using an exponential random process.

Key Insights and Ideas

  • DRL for Dynamic NFV Resource Management: The paper demonstrates the effectiveness of DRL in tackling the complex, dynamic, and large-scale resource allocation problem in NFV-enabled networks.
  • Joint Optimization of Routing and Placement: By jointly optimizing function placement and node-by-node dynamic routing, the DQN-AR algorithm can achieve better performance than methods that address these separately.
  • Handling Probabilistic Service Lifetimes: The inclusion of probabilistic service lifetimes in the model is a crucial aspect, as it reflects real-world scenarios more accurately and impacts resource availability over time.
  • Adaptive Decision-Making: The DQN-AR's ability to adapt its decisions based on the current network state and service requirements is key to its performance.
  • Trade-off between AAR and ANUC: The paper highlights the inherent trade-off between maximizing the Average Acceptance Ratio (AAR) and minimizing the Average Network Utilization Cost (ANUC), and how the wcost parameter in the reward function influences this balance.

Key Results

The simulation results indicate that the proposed DQN-AR algorithm significantly outperforms baseline algorithms (NFVdeep, Tabu search, and greedy search) in several key metrics:

  • Average Acceptance Ratio (AAR): The DQN-AR method shows an increase in the average number of admitted requests by 7% up to 20% compared to baselines.
  • Average Network Utilization Cost (ANUC): The network utilization cost decreases by 5% and 20% with the proposed DQN-AR method.
  • Performance across Network Topologies: The algorithm demonstrates good performance across various network sizes (10 to 100 nodes), with AAR generally increasing and ANUC decreasing over iterations as the agent learns.
  • Effect of wcost: Increasing the wcost parameter leads to a decrease in AAR but also a decrease in ANUC, demonstrating the ability to tune the algorithm's behavior.
  • Effect of Service Lifetime: Longer service lifetimes negatively impact AAR due to reduced resource availability, but DQN-AR adapts well to these conditions.

Limitations and Future Work

  • Computational Complexity: While DQN-AR shows good performance, the computational complexity of DNN-based algorithms can be a concern, especially with large state and action spaces. The paper acknowledges this by stating the complexity in terms of input/output/hidden layers.
  • Simulation-Based Evaluation: The results are based on simulations, and real-world deployment might introduce additional complexities.
  • Future Work: The authors suggest investigating other RL algorithms, such as Recurrent Deterministic Policy Gradient (RDPG), for proactive and predictive resource allocation. They also propose studying other RL algorithms in NFV-enabled networks.

Summary updated

BibTeX
@article{nouruzi2022online,
  author = {Ali Nouruzi and Abolfazl Zakeri and Mohammad Reza Javan and Nader Mokari and Rasheed Hussain and S. M. Ahsan Kazmi},
  title = {{Online Service Provisioning in NFV-Enabled Networks Using Deep Reinforcement Learning}},
  journal = {IEEE Trans. Netw. Serv. Manag},
  year = {2022},
  doi = {10.1109/tnsm.2022.3159670},
}
Figure