CORE: Compensable Reward as a Catalyst for Improving Offline RL in Wireless Networks

Lipeng Zu, Hansong Zhou, Yu Qian, Shayok Chakraborty, Yukun Yuan, Linke Guo, Xiaonan ZhangarXiv 2025 · DOI · arXiv

Abstract

CORE (COmpensable REwards) is a novel offline reinforcement learning framework designed to handle the noisy, suboptimal, and biased datasets typical of real-world wireless networks. By using contrastive learning to separate "expert" behaviors in a latent space and relabeling rewards based on proximity to these experts, CORE significantly improves policy reliability and generalization without requiring risky online exploration.

Introduction

Reinforcement Learning (RL) holds massive potential for managing complex wireless tasks like spectrum allocation and edge computing. However, online RL—where an agent learns by trial and error in real-time—is often too dangerous for mission-critical infrastructure. A single "bad" exploratory action could crash a network or disrupt emergency services.

Offline RL solves this by learning from pre-collected historical data. But there’s a catch: wireless data is messy. It’s often "suboptimal" (collected from mediocre policies), biased (only covers specific conditions), and suffers from "distributional shift" (e.g., a policy trained on quiet campus Wi-Fi fails in a crowded stadium). Existing offline RL methods struggle to distinguish between lucky high-reward actions and true expert behavior, leading to unstable performance when deployed.

The CORE Framework

The authors propose CORE, which introduces three main innovations to the standard offline RL pipeline to bridge the gap between noisy historical data and robust policy performance.

The CORE framework architecture: Step 1 identifies experts via clustering, Step 2 uses contrastive loss in an enhanced CVAE, and Step 3 calculates compensable rewards in the latent space.
  1. State-Based Expert Identification: Instead of looking at entire trajectories (which might be mostly bad with one lucky break), CORE uses K-means clustering to group similar states. Within each cluster, it identifies the top-performing samples as "experts." This provides more granular, context-aware guidance.
  2. Enhanced CVAE with Contrastive Loss: The framework trains a Conditional Variational Autoencoder (CVAE) to map state-action pairs into a latent space. Crucially, they add a contrastive loss function. This forces expert behaviors to cluster tightly together while pushing non-expert behaviors far away, creating a clear "boundary" in the model's internal representation.
  3. Compensable Rewards: Once the latent space is organized, CORE calculates a "compensable reward" based on how close an action is to the expert centroid in that latent space. This adjusted reward is added to the original environment reward, effectively "guiding" the agent toward expert-like decisions even when the original data is sparse or noisy.

Datasets and Evaluation

The researchers evaluated CORE across three diverse, real-world wireless use cases:

  • ORAN-Cloud Resource Allocation: Used the Alibaba Cluster Trace v2018, a massive open-source dataset of real cluster workloads, to simulate server management in Open Radio Access Networks.
  • Edge Computing Task Offloading: A custom setup using NVIDIA Orin Nano and RTX 3060 Ti hardware, running YOLOv11n vision models to simulate offloading decisions for mobile devices.
  • Wi-Fi Channel Selection: Real-world signal data collected using USRP N210 software-defined radios over seven days of continuous transmission.

Key Insights: Intra-cluster Variance

A critical discovery in this work is that expert datasets naturally exhibit higher reward variance within similar states compared to non-expert datasets. This "local reward expressiveness" is a key signal that CORE exploits to guide learning. While many RL methods focus on whole trajectories, the authors argue that in dynamic wireless environments, trajectory-based identification is unreliable due to high noise. State-based clustering provides a much more robust "anchor" for identifying high-quality data.

Reward difference across various environments, showing higher variance in expert datasets compared to non-expert datasets as the number of clusters increases.

By injecting Gaussian noise into the CVAE latent space during training, the model explores "near-expert" regions. This allows the agent to discover actions that might be even better than those in the limited offline dataset, effectively "recalibrating" the reward structure through the "Compensable" logic.

Key Results

CORE was tested against several heavyweights in the offline RL space, including CQL, BCQ, and SAC, demonstrating several advantages:

  • Superior Performance: In the ORAN-Cloud task, CORE consistently achieved the highest average returns, outperforming baselines like CQL and trajectory-based methods like CLUE.
  • Data Efficiency: In the edge offloading task, CORE showed substantial improvements even with a limited dataset of only 50,000 samples—a scenario where most offline RL algorithms fail to generalize.
  • OOD Robustness: When tested on "Shifted Workload Distributions" (e.g., training on light traffic but testing on heavy traffic), CORE maintained high success rates while baseline methods saw performance craters.
  • Hardware Validation: On the USRP Wi-Fi testbed, CORE achieved a significantly higher packet success rate (approx. 0.80) compared to standard CQL (approx. 0.76) at low latency intervals.

Limitations and Future Work

While CORE shows impressive results, performance is sensitive to the number of clusters ($k$) and the scaling factor ($ au$) for the compensable reward, which currently requires manual tuning. Additionally, training the enhanced CVAE with contrastive loss adds computational overhead. Future directions include exploring automated hyperparameter selection and extending the framework to multi-agent wireless scenarios.

Summary updated

Abstract

The operation of next-generation wireless networks increasingly hinges on Deep Reinforcement Learning (DRL) to optimize critical network decisions. Offline RL trains agents using accessible datasets and thus eliminates costly online interactions, rendering it particularly advantageous in dynamic wireless environments that demand low latency and high safety. However, its capability remains underexplored in wireless settings, where offline data are often limited and subject to partial observability. As a result, the learned policy fails to fully capture the informative decision patterns hidden in the dataset. To bridge this gap, we propose MEO, a framework that mines reliable expert signals from wireless datasets for offline RL deployment. MEO first identifies reliable expert samples from imperfect offline data, and then learns a structured latent representation to better distinguish expert and non-expert behaviors. Based on the representation, MEO further derives a compensable reward signal that helps offline RL algorithms more effectively exploit valuable expert knowledge. We evaluate MEO in three practical wireless network scenarios using datasets collected from real-world deployments. The results show that MEO consistently outperforms existing methods across all three scenarios. Notably, MEO can be seamlessly integrated with any offline RL algorithms, highlighting its generalizability and potential for wide adoption in wireless networks. Code is available at https://github.com/lpzu/MEO.
BibTeX
@misc{zu2025core,
  author = {Lipeng Zu and Hansong Zhou and Yu Qian and Shayok Chakraborty and Yukun Yuan and Linke Guo and Xiaonan Zhang},
  title = {{CORE: Compensable Reward as a Catalyst for Improving Offline RL in Wireless Networks}},
  year = {2025},
  eprint = {2512.19671},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2512.19671},
}
Figure