A Graph Convolutional Network-Based Deep Reinforcement Learning Approach for Resource Allocation in a Cognitive Radio Network

Di Zhao, Hao Qin, Bin Song, Beichen Han, Xiaojiang Du, Mohsen Guizani2020 · DOI

TL;DR

This paper tackles the challenge of efficiently allocating radio resources in cognitive radio networks (CRNs) to maximize data rates while ensuring the quality of service for primary users. They propose a novel approach that combines Graph Convolutional Networks (GCNs) with Deep Reinforcement Learning (DRL) to learn optimal resource allocation strategies by modeling the CRN as a dynamic graph.

What Problem Does This Paper Address?

The explosive growth of mobile data traffic, especially with the advent of 5G, has led to a severe spectrum scarcity. Cognitive radio (CR) technology offers a solution by allowing secondary users (SUs) to opportunistically access spectrum licensed to primary users (PUs) without causing unacceptable interference. However, effectively managing these shared resources in underlay CRNs is complex due to:

  • Interference: Co-channel interference (CCI) between PUs and SUs, and among SUs themselves, is a major challenge.
  • Dynamic Environment: User mobility and changing channel conditions make the network topology dynamic.
  • Inaccurate Channel State Information (CSI): Obtaining precise CSI is often difficult in real-world scenarios.
  • NP-hard Problem: Resource allocation in CRNs is generally an NP-hard problem, making traditional optimization methods computationally expensive and slow for real-time applications.

Existing approaches often struggle with scalability, computational complexity, and the inability to fully leverage the network's topological information.

Methods

The paper proposes a "GCN+DRL" framework for resource allocation in underlay CRNs. Here's a breakdown of their key methods:

  • Dynamic Graph Construction: The CRN is modeled as a dynamic graph where signal links are treated as vertices and interference links as edges. User mobility is simulated using a random walk model, making the graph topology dynamic. This approach simplifies the graph by focusing on interference links, which are crucial for resource allocation.
  • Graph Convolutional Network (GCN): To handle the non-Euclidean nature of graph data and extract crucial interference features, a GCN is employed. Unlike traditional CNNs, GCNs can effectively process graph-structured data. The GCN learns to represent the network's topology and interference patterns.
  • Deep Reinforcement Learning (DRL): A DRL framework is used to learn an optimal resource allocation policy. The agent (a central controller) observes the state of the CRN (represented by the dynamic graph), takes actions (channel selection and power adaptation), and receives rewards based on the achieved data rate and adherence to QoS constraints.
  • End-to-End Learning: A key innovation is the design of an end-to-end learning model. This model integrates feature extraction (via GCN) and task learning (resource allocation) into a single network. This avoids the mismatch that can occur when features are extracted separately and then fed into a task-specific model, allowing the reward signal to guide both feature representation and policy generation simultaneously.
  • State Representation: Instead of relying on accurate CSI, the state is defined by the user distance distribution and resource occupation, which are derived from the dynamic graph. This makes the system more robust to CSI inaccuracies.
  • Action Space: Actions involve both channel selection and power adaptation for SUs.

Datasets

Not discussed in the paper. The authors conducted simulations using parameters from existing literature and their own defined settings.

Key Insights and Ideas

  • Topology Matters: The paper strongly emphasizes that effectively leveraging the network's topology is crucial for optimal resource allocation, especially in dynamic environments. Traditional methods that treat network states as simple Euclidean data miss this vital information.
  • GCNs for Interference: GCNs are shown to be highly effective in extracting complex interference features from the dynamic graph representation of the CRN, which is a significant advantage over methods that don't explicitly model these relationships.
  • End-to-End Learning for Synergy: Integrating feature extraction and decision-making into a single DRL framework allows for a more cohesive learning process. The reward signal can directly influence how features are learned to best serve the allocation task, leading to better overall performance.
  • Robustness to CSI: By using user distance and resource occupation as state inputs instead of precise CSI, the proposed method becomes more practical and resilient to real-world limitations.

Key Results

  • The proposed GCN+DRL algorithm significantly outperforms random strategies, a standard policy gradient algorithm, and a CNN-based DRL approach in terms of achievable data rate.
  • The GCN+DRL algorithm achieves a stable data rate of approximately 3100 kbps, outperforming other methods.
  • The convergence time for the GCN+DRL algorithm is reported as 8800 seconds, which is considerably faster than the PG algorithm (17,000 seconds) and the CNN+DRL (≥30,880 seconds).
  • The algorithm demonstrates stable convergence performance, with the total loss dropping to a minimum after about 5000 iterations.

Limitations and Future Work

  • Scalability: While the paper claims the GCN approach is scalable, the specific computational complexity is noted as O((NM)(U+SV)), which could still be a concern for extremely large networks.
  • Centralized Agent: The current framework uses a centralized controller as the agent. Future work could explore decentralized or multi-agent DRL approaches for greater scalability and robustness.
  • User Mobility Simulation: The random walk model for user mobility is a simplification. More realistic mobility models could be investigated.
  • Interference Temperature: The paper mentions using interference temperature from literature but doesn't detail its derivation or impact on the model's sensitivity.

The authors suggest that their end-to-end learning model enhances cohesion and adaptation to practical problems, implying that this approach is well-suited for real-world CRN deployments.

Summary updated

BibTeX
@misc{zhao2020graph,
  author = {Di Zhao and Hao Qin and Bin Song and Beichen Han and Xiaojiang Du and Mohsen Guizani},
  title = {{A Graph Convolutional Network-Based Deep Reinforcement Learning Approach for Resource Allocation in a Cognitive Radio Network}},
  year = {2020},
  doi = {10.3390/s20185216},
}
Figure