Offline Reinforcement Learning for Wireless Network Optimization with Mixture Datasets

Kun Yang, Cong Shen, Jing Yang, Shu-Ping Yeh, Jerry Sydir

TL;DR

This paper tackles the challenge of applying reinforcement learning (RL) to wireless network optimization without the need for risky online exploration. They propose a novel offline RL approach that combines data from multiple, potentially suboptimal, network control policies to achieve near-optimal performance, even when individual data sources are weak.

What Problem Does This Paper Address?

Wireless networks, particularly radio resource management (RRM), are complex and dynamic systems that could benefit from RL. However, traditional online RL algorithms require direct interaction with the live network to learn. This "exploration" phase can lead to temporary performance degradation and poor Quality of Service (QoS) for users, which is unacceptable for network operators. This paper addresses this by exploring offline RL, which learns from pre-collected data, and proposing a method to make it effective even when the data comes from less-than-ideal network control policies.

Methods

The authors first evaluate existing state-of-the-art offline RL algorithms – Behavior Constrained Q-learning (BCQ), Conservative Q-learning (CQL), and Implicit Q-learning (IQL) – on a wireless user scheduling problem. They find that the performance of these algorithms is highly dependent on the quality of the data collected by the "behavior policy" (the policy that generated the data). To overcome this limitation, they propose a novel approach: mixture datasets. This involves combining data collected by multiple different behavior policies, even if those policies are individually suboptimal. The intuition is that a diverse set of data, even from weak sources, can collectively provide enough information to learn a good policy.

Datasets

The paper uses datasets generated by four rule-based behavior policies (Random, Greedy, Time Division Multiplexing (TDM), and ITLinQ) and two online RL policies (one early-stopped and one well-trained). These datasets are used to train and evaluate the offline RL algorithms. The authors do not introduce new benchmark datasets but rather use existing policies to generate data for their specific RRM problem.

Key Insights and Ideas

  • Offline RL is promising but data-hungry: Offline RL can avoid the risks of online exploration, but its performance is critically tied to the quality of the data it learns from.
  • Suboptimal data doesn't mean suboptimal results: Combining data from multiple, even poor, behavior policies can lead to a near-optimal offline RL policy. This is because the mixture can cover a wider range of states and actions than any single policy could.
  • Diversity is key: The success of the mixture dataset approach hinges on the diversity of the behavior policies used to collect the data.

Key Results

  • When trained on data from a single, suboptimal behavior policy, offline RL algorithms (specifically IQL) showed only modest performance gains over the behavior policy itself.
  • By mixing datasets from multiple behavior policies, the proposed offline RL approach achieved near-optimal performance, closely matching the performance of a well-trained online RL agent. For example, "Mixed-RL" (combining an early-stopped online RL policy with rule-based policies) achieved over 95% of the optimal online RL performance.
  • The mixture dataset approach significantly outperformed any individual behavior policy used for data collection.

Limitations and Future Work

The paper acknowledges that their proposed offline RL solution is centralized, which may not scale well to very large wireless networks. They suggest that developing a multi-agent offline RL solution would be an interesting future research direction. Additionally, the paper mentions the potential for further enhancements, such as an ensemble-based policy to improve training efficiency and an offline-to-online strategy for seamless adaptation to new environments, though these are not the primary focus of this specific paper.

Summary updated

Abstract

The recent development of reinforcement learning (RL) has boosted the adoption of online RL for wireless radio resource management (RRM). However, online RL algorithms require direct interactions with the environment, which may be undesirable given the potential performance loss due to the unavoidable exploration in RL. In this work, we first investigate the use of \emph{offline} RL algorithms in solving the RRM problem. We evaluate several state-of-the-art offline RL algorithms, including behavior constrained Q-learning (BCQ), conservative Q-learning (CQL), and implicit Q-learning (IQL), for a specific RRM problem that aims at maximizing a linear combination {of sum and} 5-percentile rates via user scheduling. We observe that the performance of offline RL for the RRM problem depends critically on the behavior policy used for data collection, and further propose a novel offline RL solution that leverages heterogeneous datasets collected by different behavior policies. We show that with a proper mixture of the datasets, offline RL can produce a near-optimal RL policy even when all involved behavior policies are highly suboptimal.
BibTeX
@article{yang2023offline,
  author = {Kun Yang and Cong Shen and Jing Yang and Shu-Ping Yeh and Jerry Sydir},
  title = {{Offline Reinforcement Learning for Wireless Network Optimization with Mixture Datasets}},
  journal = {IEEE Trans. Wireless Communications},
  year = {2023},
  url = {https://openreview.net/forum?id=NMrRFsiOhF},
}
Figure