Abstract
CORE (COmpensable REwards) is a novel offline reinforcement learning framework designed to handle the noisy, suboptimal, and biased datasets typical of real-world wireless networks. By using contrastive learning to separate "expert" behaviors in a latent space and relabeling rewards based on proximity to these experts, CORE significantly improves policy reliability and generalization without requiring risky online exploration.
Introduction
Reinforcement Learning (RL) holds massive potential for managing complex wireless tasks like spectrum allocation and edge computing. However, online RL—where an agent learns by trial and error in real-time—is often too dangerous for mission-critical infrastructure. A single "bad" exploratory action could crash a network or disrupt emergency services.
Offline RL solves this by learning from pre-collected historical data. But there’s a catch: wireless data is messy. It’s often "suboptimal" (collected from mediocre policies), biased (only covers specific conditions), and suffers from "distributional shift" (e.g., a policy trained on quiet campus Wi-Fi fails in a crowded stadium). Existing offline RL methods struggle to distinguish between lucky high-reward actions and true expert behavior, leading to unstable performance when deployed.
The CORE Framework
The authors propose CORE, which introduces three main innovations to the standard offline RL pipeline to bridge the gap between noisy historical data and robust policy performance.
- State-Based Expert Identification: Instead of looking at entire trajectories (which might be mostly bad with one lucky break), CORE uses K-means clustering to group similar states. Within each cluster, it identifies the top-performing samples as "experts." This provides more granular, context-aware guidance.
- Enhanced CVAE with Contrastive Loss: The framework trains a Conditional Variational Autoencoder (CVAE) to map state-action pairs into a latent space. Crucially, they add a contrastive loss function. This forces expert behaviors to cluster tightly together while pushing non-expert behaviors far away, creating a clear "boundary" in the model's internal representation.
- Compensable Rewards: Once the latent space is organized, CORE calculates a "compensable reward" based on how close an action is to the expert centroid in that latent space. This adjusted reward is added to the original environment reward, effectively "guiding" the agent toward expert-like decisions even when the original data is sparse or noisy.
Datasets and Evaluation
The researchers evaluated CORE across three diverse, real-world wireless use cases:
- ORAN-Cloud Resource Allocation: Used the Alibaba Cluster Trace v2018, a massive open-source dataset of real cluster workloads, to simulate server management in Open Radio Access Networks.
- Edge Computing Task Offloading: A custom setup using NVIDIA Orin Nano and RTX 3060 Ti hardware, running YOLOv11n vision models to simulate offloading decisions for mobile devices.
- Wi-Fi Channel Selection: Real-world signal data collected using USRP N210 software-defined radios over seven days of continuous transmission.
Key Insights: Intra-cluster Variance
A critical discovery in this work is that expert datasets naturally exhibit higher reward variance within similar states compared to non-expert datasets. This "local reward expressiveness" is a key signal that CORE exploits to guide learning. While many RL methods focus on whole trajectories, the authors argue that in dynamic wireless environments, trajectory-based identification is unreliable due to high noise. State-based clustering provides a much more robust "anchor" for identifying high-quality data.
By injecting Gaussian noise into the CVAE latent space during training, the model explores "near-expert" regions. This allows the agent to discover actions that might be even better than those in the limited offline dataset, effectively "recalibrating" the reward structure through the "Compensable" logic.
Key Results
CORE was tested against several heavyweights in the offline RL space, including CQL, BCQ, and SAC, demonstrating several advantages:
- Superior Performance: In the ORAN-Cloud task, CORE consistently achieved the highest average returns, outperforming baselines like CQL and trajectory-based methods like CLUE.
- Data Efficiency: In the edge offloading task, CORE showed substantial improvements even with a limited dataset of only 50,000 samples—a scenario where most offline RL algorithms fail to generalize.
- OOD Robustness: When tested on "Shifted Workload Distributions" (e.g., training on light traffic but testing on heavy traffic), CORE maintained high success rates while baseline methods saw performance craters.
- Hardware Validation: On the USRP Wi-Fi testbed, CORE achieved a significantly higher packet success rate (approx. 0.80) compared to standard CQL (approx. 0.76) at low latency intervals.
Limitations and Future Work
While CORE shows impressive results, performance is sensitive to the number of clusters ($k$) and the scaling factor ($ au$) for the compensable reward, which currently requires manual tuning. Additionally, training the enhanced CVAE with contrastive loss adds computational overhead. Future directions include exploring automated hyperparameter selection and extending the framework to multi-agent wireless scenarios.
Summary updated