Hybrid Centralized-Distributed Resource Allocation Based on Deep Reinforcement Learning for Cooperative D2D Communications

Yang Yu, Xiaoqing Tang

TL;DR

This paper introduces a hybrid centralized-distributed resource allocation scheme for cooperative device-to-device (D2D) communications. By combining Deep Reinforcement Learning (DRL) for local power and spectrum optimization with the Kuhn-Munkres (KM) algorithm for global link matching, the authors achieve near-optimal energy efficiency with significantly lower signaling overhead than traditional centralized methods.

What Problem Does This Paper Address?

As 5G and 6G networks strive for "green communication," improving spectrum and energy efficiency is paramount. Cooperative D2D communication offers a "win-win": D2D users (DUs) act as relays for cellular users (CUs) with poor connections, and in exchange, DUs get exclusive access to cellular spectrum.

However, optimizing this is a nightmare. It requires joint coordination of spectrum allocation, power control, and link matching (deciding which DU pairs with which CU). Mathematically, this is a Mixed Integer Nonlinear Programming (MINLP) problem that is NP-hard. Existing centralized solutions require massive amounts of global channel state information (CSI), leading to high signaling overhead, while purely distributed solutions often struggle with slow convergence and coordination conflicts.

Methods

The authors propose a two-stage hybrid architecture that decouples the problem:

  1. Distributed DRL (Lower Level): Each D2D transmitter acts as an independent agent using a Deep Q-Network (DQN). These agents autonomously optimize spectrum allocation and power control for potential CU-DU pairs using only local information. To handle constraints (like minimum Quality of Service), the authors use "reward shaping"—penalizing the agent if it fails to meet required thresholds.
  2. Centralized KM Algorithm (Upper Level): Once the DRL agents calculate their potential "utility" (Weighted Sum Energy Efficiency), they report these values to the Base Station (BS). The BS then uses the Kuhn-Munkres (KM) algorithm—a classic bipartite graph matching tool—to find the global optimal matching between all CUs and DUs.
  3. Cooperative Link Sets: To further reduce complexity, the authors introduce a heuristic to limit the number of CUs a DU considers. Instead of looking at every CU in the cell, a DU only considers those within a specific "cooperative range," making the utility matrix sparse and reducing signaling.

Datasets

The study does not use a public dataset; instead, it relies on Monte Carlo simulations to validate the model.

  • Environment: A single-cell network with a 500m radius.
  • Parameters: 10 CUs and 5–15 D2D pairs.
  • Channel Model: A simplified path loss model with an exponent of 3.8, assuming static or quasi-static channel conditions dominated by large-scale fading.

Key Insights and Ideas

  • WSEE over EE: The authors use Weighted Sum Energy Efficiency (WSEE) as the metric. Unlike standard Energy Efficiency, WSEE allows network operators to assign different priorities to users, ensuring fairness or favoring high-priority cellular links while still optimizing for the whole system.
  • The Power of Decoupling: By breaking the MINLP problem into a local optimization task (DRL) and a matching task (KM), the authors bypass the "curse of dimensionality" that usually plagues DRL in large-scale networks.
  • Hybrid is the "Sweet Spot": The hybrid approach captures the best of both worlds: the autonomy and low latency of distributed learning and the global coordination of centralized matching.

Key Results

  • Near-Optimal Performance: The proposed scheme achieves over 95% of the performance of an exhaustive search (the theoretical upper bound) but with a fraction of the computational cost.
  • Convergence Speed: The DRL agents converge quickly, with WSEE values stabilizing after approximately 110 episodes.
  • Efficiency Gains: Compared to random resource allocation, the proposed method shows a massive improvement in WSEE as the number of D2D links increases.
  • Reduced Overhead: Using "cooperative link sets" reduced the number of active agents and signaling exchanges significantly while still maintaining roughly 85% of the optimal WSEE.

Limitations and Future Work

The primary limitation is the assumption of static or quasi-static channels. In highly mobile environments (like V2X), channel conditions change rapidly, which might require more frequent retraining of the DQN or a more robust way to handle CSI delays.

The authors suggest that future work will focus on further enhancing the efficiency of the matching process and exploring how this model performs in more complex, ultra-dense heterogeneous network scenarios.

Summary updated

Abstract

Device-to-device (D2D) technology enables direct communication between adjacent devices within cellular networks. Due to its high data rate, low latency, and performance improvement in spectrum and energy efficiency, it has been widely investigated and applied as a critical technology in 5G New Radio (NR). In addition to conventional overlay and underlay D2D communications, cooperative D2D communication, which can achieve a win-win situation between cellular users (CUs) and D2D users (DUs) through cooperative relaying technique, has attracted extensive attention from academic and industrial circles in the past decade. This paper delves into optimizing joint spectrum allocation, power control, and link-matching between multiple CUs and DUs for cooperative D2D communications, using weighted sum energy efficiency (WSEE) as the performance metric to address the challenges of green communication and sustainable development. This integer programming problem can be decomposed into a classic weighted bipartite graph matching and a series of nonconvex spectrum allocation and power control problems between potentially matched cellular and D2D link pairs. To address this issue, we propose a hybrid centralized-distributed scheme based on deep reinforcement learning (DRL) and the Kuhn-Munkres (KM) algorithm. Leveraging the latter, the CUs and DUs autonomously optimize spectrum allocation and power control by only utilizing local information. Then, the base station (BS) determines the link matching. Simulation results reveal that it achieves near-optimal performance and significantly enhances the network convergence speed with low signaling overheads. In addition, we also propose and utilize cooperative link sets for corresponding D2D links to accelerate the proposed scheme and reduce signaling exchange further.
BibTeX
@misc{yu2024hybrid,
  author = {Yang Yu and Xiaoqing Tang},
  title = {{Hybrid Centralized-Distributed Resource Allocation Based on Deep Reinforcement Learning for Cooperative D2D Communications}},
  year = {2024},
  eprint = {2410.03177},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2410.03177},
}
Figure