FlagVNE: A Flexible and Generalizable Reinforcement Learning Framework for Network Resource Allocation

Tianfu Wang, Qilin Fan, Chao Wang, Long Yang, L. K. Ding

Abstract

Virtual Network Embedding (VNE) is a fundamental challenge in network virtualization, requiring the efficient mapping of virtual network requests (VNRs) onto shared physical infrastructure. Existing reinforcement learning (RL) solutions often suffer from rigid, unidirectional node mapping sequences and poor generalization across varying network scales. We present FlagVNE, a novel framework that addresses these limitations through a bidirectional action-based Markov Decision Process (MDP) and a hierarchical policy architecture. By jointly selecting virtual and physical nodes, FlagVNE expands the search space for optimal embeddings. A hierarchical policy architecture manages the resulting action space, while a curriculum-based meta-learning approach ensures robust performance across different network sizes. Experimental evaluations on standard topologies demonstrate that FlagVNE achieves significant improvements in request acceptance rates and long-term revenue compared to state-of-the-art baselines.

Introduction

In the era of 5G and cloud computing, Network Virtualization (NV) enables multiple isolated virtual networks to coexist on a common physical substrate. The core efficiency of this paradigm hinges on the Virtual Network Embedding (VNE) problem: mapping virtual nodes and links to physical counterparts while satisfying resource constraints such as CPU, storage, and bandwidth.

VNE is an NP-hard combinatorial problem. While traditional heuristics and recent RL-based methods have been proposed, they typically follow a "unidirectional" mapping order—fixing the sequence of virtual nodes before placement. This rigidity often leads to sub-optimal solutions. Furthermore, models trained on specific network sizes frequently fail to generalize to larger or more complex requests.

Illustration of Virtual Network Embedding (VNRs) mapping onto a Physical Network.

The FlagVNE Framework

FlagVNE introduces three primary innovations to overcome the limitations of existing VNE solvers.

Bidirectional Action-based MDP

Unlike unidirectional approaches, FlagVNE employs a bidirectional action design. At each step, the agent simultaneously selects a virtual node and a physical node. This approach allows the agent to dynamically prioritize the most critical virtual nodes based on the current state of the physical network, significantly expanding the search space and improving the potential for finding optimal mappings.

Hierarchical Policy Architecture

To handle the increased complexity of joint node selection, FlagVNE utilizes a hierarchical decoder. A high-level ordering policy identifies the next virtual node, while a low-level placement policy determines the physical host. This architecture reduces the action space complexity from multiplicative to additive, ensuring computational efficiency.

The FlagVNE training and policy architecture, featuring meta-learning and a hierarchical GNN-based encoder-decoder.

Meta-RL with Curriculum Scheduling

To achieve broad generalization, FlagVNE incorporates Model-Agnostic Meta-Learning (MAML). This allows the model to learn a "meta-policy" that can be rapidly adapted to size-specific sub-policies. To stabilize training, a curriculum scheduling strategy is introduced: the agent is first trained on smaller, simpler VNRs and gradually progresses to larger tasks as its confidence (measured by policy entropy) increases.

Experimental Results

The framework was evaluated on the GEANT and WX100 physical network benchmarks using 1,000 Poisson-arriving VNRs. FlagVNE demonstrated superior performance across all key metrics:

  • Acceptance Rate: Surpassed the best RL baseline (A3C-GCN) by 10.4% on the GEANT topology.
  • Revenue: Improved Long-term Average Revenue (LAR) by up to 12.8% on the WX100 topology.
  • Resource Efficiency: Achieved a Revenue-to-Cost (R2C) ratio gain of up to 45.1% over traditional heuristics.

Conclusion

FlagVNE represents a significant step forward in automated network management. By combining bidirectional decision-making with meta-learning and curriculum strategies, it provides a flexible and highly generalizable solution for the VNE problem. Future work will focus on integrating link routing directly into the RL process and enhancing scalability for ultra-large-scale data center networks.

Summary updated

BibTeX
@inproceedings{wang2024flagvne,
  author = {Tianfu Wang and Qilin Fan and Chao Wang and Long Yang and L. K. Ding},
  title = {{FlagVNE: A Flexible and Generalizable Reinforcement Learning Framework for Network Resource Allocation}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2024},
  url = {https://openreview.net/forum?id=t6q3DrwAaR},
}
Figure