Structured Reinforcement Learning for Delay-Optimal Data Transmission in Dense mmWave Networks

Shufan Wang, Guojun Xiong, Shichen Zhang, Huacheng Zeng, Jian Li, Shivendra S. Panwar

TL;DR

This paper introduces mmDPT-TS, a structured reinforcement learning (RL) algorithm designed to minimize transmission delays in dense millimeter-wave (mmWave) networks. By modeling the system as a "Restless Multi-Armed Bandit with Fairness" (RMAB-F), the authors develop a low-complexity index policy that bypasses the "curse of dimensionality" typical in large-scale wireless networks, achieving near-optimal performance with sub-linear regret.

What Problem Does This Paper Address?

Millimeter-wave (mmWave) technology is a pillar of 5G and 6G, offering massive bandwidth. However, its high frequency makes it extremely sensitive to blockages and mobility, leading to intermittent connectivity. To solve this, researchers propose "dense cell-free networks" where many Access Points (APs) serve a population of users.

The core challenge is Data Packet Transmission (mmDPT): when a user generates a request, which AP should handle it?

  1. Unreliable Downlinks: Wireless channels are time-varying and unpredictable.
  2. Limited Capacity: APs have finite service rates.
  3. Fairness Constraints: Resources must be shared fairly among users.
  4. Computational Complexity: In a dense network, the number of possible states (queue lengths across all AP-user pairs) grows exponentially. Standard RL algorithms (like DQN or standard Thompson Sampling) struggle to converge or require massive computational power to solve complex Bellman equations in real-time.

Methods

The authors propose a Structured Reinforcement Learning approach that exploits the mathematical structure of the problem rather than treating it as a "black box."

  • RMAB-F Formulation: They model the system as a Restless Multi-Armed Bandit problem with a fairness constraint. Unlike standard bandits, "restless" arms change state even when not played.
  • mmDPT Index Policy: Instead of solving the full MDP (Markov Decision Process), they design a low-complexity index policy. They use Linear Programming (LP) based on "occupancy measures" to calculate an index for each user-AP pair. This index represents the "value" of sending a request to a specific AP.
  • mmDPT-TS (Thompson Sampling): The RL layer uses Thompson Sampling to learn the unknown transition probabilities (channel qualities). Crucially, instead of solving a massive MDP at each step, it samples parameters and then applies the structured index policy.
  • Asymptotic Optimality: The authors provide a theoretical proof showing that as the network scales (more users and APs), their policy approaches the theoretical optimal performance.

Datasets

The study uses two primary data sources for evaluation:

  1. 60GHz mmWave Testbed: A custom-built hardware setup using Software Defined Radios (USRP X310) and ADI 60GHz RF frontends. They collected 468,750 Error Vector Magnitude (EVM) samples in a conference room setting (4 APs, 30 users) to create realistic traces of channel volatility.
  2. Synthetic Traces: Used to test the algorithm's scalability up to 100 users. These traces used Poisson arrival processes for data requests and arithmetic sequences for packet delivery probabilities.

Key Insights and Ideas

  • Structure vs. Brute Force: The "aha moment" is that you don't need to solve the entire state space if you can prove the problem is "indexable." By focusing on the index policy, the algorithm operates in a much smaller dimensional subspace.
  • Decoupling Constraints: The authors successfully coupled scheduling (which AP to use) with fairness (resource limits) by sorting requests by their calculated indices and assigning them greedily until capacity is reached.
  • The "Global Attractor": The paper verifies numerically that the system state converges to a stable equilibrium (global attractor), which is a necessary condition for their asymptotic optimality proof to hold in practice.

Key Results

  • Regret Bounds: The algorithm achieves a Bayesian regret of $O(\sqrt{T} \log T)$, which is the theoretical "gold standard" for online learning efficiency.
  • Delay Reduction: In both testbed and synthetic traces, mmDPT-TS significantly outperformed baselines like Whittle Indexing, DQN, and Soft Actor-Critic (SAC) in terms of accumulated delay (regret).
  • Computational Efficiency: mmDPT-TS is much faster than traditional index-based methods (like TS-Whittle) because it avoids iterative value iteration, making it feasible for real-time implementation in 5G/6G hardware.
  • Scalability: The "optimality gap" (the difference between the algorithm and the theoretical best) effectively closed to zero as the network density increased.

Limitations and Future Work

  • Centralized Control: The current model assumes a centralized controller manages all APs. In massive-scale deployments, a decentralized or federated version might be necessary to reduce signaling overhead.
  • Uplink Assumptions: The paper assumes the uplink (user to AP request) is perfectly reliable. While reasonable for small control packets, real-world interference might complicate this.
  • Global Attractor Proof: While verified numerically, a rigorous analytical proof of the "global attractor" property for this specific RMAB-F remains an open mathematical challenge.
  • Mobility: While the model handles time-varying channels, explicit modeling of high-speed user mobility (e.g., in vehicular networks) could be a promising future direction.

Summary updated

Abstract

We study the data packet transmission problem (mmDPT) in dense cell-free millimeter wave (mmWave) networks, i.e., users sending data packet requests to access points (APs) via uplinks and APs transmitting requested data packets to users via downlinks. Our objective is to minimize the average delay in the system due to APs’ limited service capacity and unreliable wireless channels between APs and users. This problem can be formulated as a restless multi-armed bandits problem with fairness constraint (RMAB-F). Since finding the optimal policy for RMAB-F is intractable, existing learning algorithms are computationally expensive and not suitable for practical dynamic dense mmWave networks. In this paper, we propose a structured reinforcement learning (RL) solution for mmDPT by exploiting the inherent structure encoded in RMAB-F. To achieve this, we first design a low-complexity and provably asymptotically optimal index policy for RMAB-F. Then, we leverage this structure information to develop a structured RL algorithm called mmDPT-TS, which provably achieves an $\tilde {\mathcal {O}}(\sqrt {T})$ Bayesian regret. More importantly, mmDPT-TS is computation-efficient and thus amenable to practical implementation, as it fully exploits the structure of index policy for making decisions. Extensive emulation based on data collected in realistic mmWave networks demonstrate significant gains of mmDPT-TS over existing approaches.
BibTeX
@article{wang2024structured,
  author = {Shufan Wang and Guojun Xiong and Shichen Zhang and Huacheng Zeng and Jian Li and Shivendra S. Panwar},
  title = {{Structured Reinforcement Learning for Delay-Optimal Data Transmission in Dense mmWave Networks}},
  journal = {IEEE Trans. Wireless Communications},
  year = {2024},
  url = {https://openreview.net/forum?id=0DoxXBFo7g},
}

Loading related content...

Figure