COOPO: Cyclic Offline-Online Policy Optimization Algorithm

Qisai Liu, Zhanhong Jiang, Joshua Russell Waite, Aditya Balu, Cody Fleming, Soumik SarkararXiv 2026 · arXiv

TL;DR

COOPO (Cyclic Offline-Online Policy Optimization) addresses the issue of "catastrophic forgetting" in hybrid reinforcement learning by repeatedly cycling between constrained offline training and online fine-tuning. By periodically anchoring the policy back to a static dataset and freezing the critic during online exploration, COOPO dramatically improves sample efficiency and achieves state-of-the-art performance on standard benchmarks.

What Problem Does This Paper Address?

Bridging offline and online Reinforcement Learning (RL) is a natural goal: offline RL safely extracts knowledge from pre-collected datasets, while online RL allows an agent to interact with the environment to optimize its policy. However, standard "offline-to-online" hybrid methods suffer from a major flaw: catastrophic forgetting.

When an offline-pretrained agent transitions to online fine-tuning, the sudden shift in data distribution often causes the agent's performance to collapse. The valuable knowledge acquired offline is overwritten by noisy online exploration before the agent can successfully adapt. The authors of this paper tackle this dilemma, asking: Can we design an offline-to-online framework that eliminates catastrophic forgetting while minimizing expensive online environment interactions?

Methods

Instead of treating the offline-to-online transition as a one-way street, the authors propose COOPO (Cyclic Offline-Online Policy Optimization), a framework that continuously loops between two distinct phases:

  1. Offline Phase (Dataset Anchoring): The agent trains on the static dataset using a modified version of Advantage-Weighted Actor-Critic (AWAC). Crucially, the authors introduce an explicit Kullback-Leibler (KL) divergence penalty to enforce a strict trust region, keeping the updated policy close to the previous cycle's policy. They also incorporate an explicit value function ($V$) model to ensure a smooth mathematical transition to the online phase.
  2. Online Phase (Stable Exploration): The agent interacts with the environment using an exploratory policy gradient method like Proximal Policy Optimization (PPO).
Fig. 1: Cyclic offline-online policy optimization: Traditional methods pre-train offline and then fine-tune online, but a single offline phase often leads to catastrophic forgetting and distributional shift. While warm-up phases can partially mitigate this, they fall short in preserving long-term stability. COOPO addresses this by leveraging cyclic synergy between offline re-training and online fine-tuning, repeatedly anchoring the policy to the offline dataset to enhance robustness and prevent forgetting.

The Secret Sauce: To maintain stability across these cycles, COOPO freezes the offline-trained Q-function during the online fine-tuning phase. The authors discovered that updating the critic online introduces severe value drift due to the distribution mismatch between offline and online states. Freezing it ensures a stable baseline for calculating advantages, preventing policy collapse.

Fig. 2: Schematic Diagram of COOPO: COOPO cyclically alternates KL-regularized offline training and trust-region online fine-tuning to eliminate distribution drift and catastrophic forgetting. When there is only one cycle, COOPO degenerates to vanilla offline-to-online RL.

Datasets

The framework was evaluated using the widely recognized D4RL benchmark suite, focusing on standard MuJoCo locomotion environments:

  • HalfCheetah
  • Hopper
  • Walker2d

Each environment was tested across three dataset quality tiers: medium, medium-replay, and medium-expert.

Key Insights and Ideas

  • Control-Theoretic Stability: From a control perspective, COOPO acts like a closed-loop adaptive control mechanism. The repeated offline phases act as a corrective stabilizing term, bounding policy divergence and ensuring discrete-time stability in the policy space.
  • Amortized Offline Data Cost: Theoretically, because the static dataset is reused across multiple cycles, the offline sample complexity becomes independent of the dataset size ($|D|$).
  • Superior Horizon Scaling: Sample complexity analysis reveals that COOPO scales with the effective horizon as $\tilde{O}(H^3)$ compared to pure online PPO's $\tilde{O}(H^4)$, making it exceptionally well-suited for long-horizon tasks.
  • The Power of the Loop: Periodically pulling the agent back to high-quality offline data prevents it from getting trapped in poor local optima discovered during unconstrained online exploration.

Key Results

  • Benchmark Superiority: COOPO achieved the highest reward values on 7 out of 9 D4RL tasks, outperforming prominent baselines such as Cal-QL, AWAC, Uni-O4, and IQL.
  • Massive Sample Efficiency Gains: COOPO cut total environment interactions by approximately 50% compared to pure online PPO while yielding significantly higher final returns.
  • Smooth Transitions: Empirical training curves verified that COOPO successfully mitigates the initial performance drops typical of offline-to-online transitions, showing steady, monotonic improvements across cycles.

Limitations and Future Work

Despite its strong performance, the authors note several limitations:

  • Dataset Quality Dependency: COOPO is still bounded by the coverage of the offline dataset. If the static data is poorly curated, the policy can still get trapped in local optima.
  • Sparse Rewards: The brief online exploration bursts can struggle to discover novel strategies in sparse-reward environments.
  • Computational Wall-Clock Time: Repeatedly switching back to offline retraining adds computational overhead.
  • Static Offline Buffer: Currently, online experiences are not dynamically appended back into the offline dataset for subsequent offline phases. The authors flag incorporating replayed online samples into the offline phase as a promising direction for future work.

Summary updated

Abstract

Offline reinforcement learning struggles with distributional shift and constrained performance due to static dataset limitations, while online RL demands prohibitive environment interactions. The recent advent of hybrid offline-to-online methods bridges these domains but suffers from distribution drift during transitions and catastrophic forgetting of offline knowledge. We introduce COOPO (Cyclic Offline-Online Policy Optimization), a generalized framework that repeatedly cycles between constrained offline training and online fine-tuning. Each cycle first anchors the policy to the dataset via KL-regularized advantage-weighted offline updates to minimize distributional shift and then fine-tunes it online using any policy optimization for stable exploration. Crucially, periodically returning to offline training eliminates forgetting and drift while maximizing dataset reuse. The cyclic behavior also helps reduce the online environment interactions. Theoretically, COOPO achieves better online sample efficiency, surpassing pure online RL, with guaranteed monotonic improvement under standard coverage assumptions. Extensive D4RL benchmarks demonstrate COOPO reduces online interactions versus state-of-the-art hybrids while improving final returns, maintaining robustness across diverse offline algorithms and online optimizers. This looped synergy sets new efficiency and performance standards for adaptive RL.
BibTeX
@misc{liu2026coopo,
  author = {Qisai Liu and Zhanhong Jiang and Joshua Russell Waite and Aditya Balu and Cody Fleming and Soumik Sarkar},
  title = {{COOPO: Cyclic Offline-Online Policy Optimization Algorithm}},
  year = {2026},
  eprint = {2605.18675},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2605.18675},
}

Loading related content...

Figure