Towards Optimism-Pessimism Trade-off in Model-based Offline-to-Online Reinforcement Learning

Guochen Zhou, Yijun Yang, Qiqi Duan, Qing Su, Weiming Ou, Li Shen, Shihao Ji, Yuhui Shi, Zhisong ZhangarXiv 2026 · arXiv

Moving from a static offline dataset to real-world online interaction is one of the biggest hurdles in reinforcement learning (RL). A new paper from researchers at the City University of Hong Kong and several partner institutions introduces HiP3 (Hierarchical Pareto Policy Pool), a framework that reframes how we handle the delicate balance between exploration and safety during this transition.

TL;DR

HiP3 addresses the "distribution shift" problem in offline-to-online RL by maintaining a diverse pool of policies that span the entire spectrum of optimism and pessimism. By using a contextual bandit to dynamically select the best policy for the current state during online fine-tuning, it achieves state-of-the-art adaptation on both familiar and completely novel tasks.

What Problem Does This Paper Address?

In Offline-to-Online (O2O) RL, an agent starts with a pre-collected dataset (offline) and then fine-tunes its behavior through live interaction (online). The core challenge is distribution shift: the online environment often looks different from the offline data.

To handle this, researchers usually balance two conflicting goals:

  1. Optimism: Seeking out high-reward areas (essential for exploration).
  2. Pessimism: Staying within "safe" regions where the offline data is plentiful (essential to avoid catastrophic failures).

Standard methods typically squash these into a single objective (e.g., $Reward - \lambda \cdot Uncertainty$). However, the authors discovered a "crossover" effect: optimistic policies adapt better to novel tasks, while pessimistic policies perform better on tasks identical to the offline data. Committing to a single trade-off ($\lambda$) before seeing the online task is a recipe for sub-optimal performance.

The Optimism-Pessimism Crossover: Optimistic policies (P1) excel on novel tasks, while pessimistic ones (P4) win on familiar ones.

Methods

HiP3 avoids committing to a single trade-off by building a Hierarchical Pareto Policy Pool.

  1. MOSAIC (Multi-Objective Soft Actor-critIC): This is the engine used during the offline stage. Instead of one policy, it trains a set of policies targeting different regions of the "Pareto front"—the set of all optimal trade-offs where you can't improve optimism without sacrificing pessimism. It uses "reference vectors" to guide policies toward specific balances and a neighborhood search to ensure the pool is diverse.
  2. Contextual Selection (LinUCB): Once the agent moves online, it doesn't just pick one policy and stick with it. It uses a contextual bandit algorithm (LinUCB) as a high-level controller. At every step, the controller looks at the current state and selects the most promising policy from the pool to execute and fine-tune.
  3. Dyna-style Fine-tuning: The selected policy is updated using both real online transitions and short rollouts from a learned world model.
HiP3 Pipeline: From offline Pareto pool generation to online state-dependent selection.

Datasets

The authors evaluated HiP3 on several continuous control benchmarks:

  • D4RL MuJoCo: Standard locomotion tasks (HalfCheetah, Hopper, Walker2d) with varying data quality (Random, Medium, Expert).
  • Novel Online Tasks: The researchers created harder versions of the D4RL tasks by modifying gravity, friction, noise, and reward structures to simulate real-world distribution shifts.
  • Generalization Tasks: Adroit (high-dimensional hand manipulation) and Maze2D (navigation).

Key Insights and Ideas

  • The "Aha" Moment: The realization that the "optimal" amount of pessimism isn't a global constant, but varies by task and even by state. A pessimistic policy might be best for a known hallway, but an optimistic one is needed to explore a new room.
  • Selection as Adaptation: HiP3 treats policy selection as a first-class citizen of the adaptation process. By deferring the choice of trade-off to the online stage, the agent can "feel out" which strategy works best for the specific environment it finds itself in.
  • Beyond Linear Scalarization: Most RL methods use linear combinations of objectives. HiP3 argues (and shows) that if the objective space is non-convex, linear methods will miss critical trade-off points that MOSAIC can find.

Key Results

HiP3 demonstrated significant performance gains over representative O2O baselines like Cal-QL, IQL, and AWAC.

  • Aggregate Performance: HiP3 achieved the highest average normalized scores across both "same" and "novel" task categories.
  • Novel Task Superiority: On novel tasks, HiP3's advantage was most pronounced, outperforming the next best baseline (MBPO) by a wide margin (e.g., 77.3 vs 59.4 total mean score).
  • Robustness: Even under "Stronger Distribution Shifts" (extreme changes to physics), HiP3 maintained its lead, while other methods' performances collapsed.
Learning curves showing HiP3 (pink) consistently reaching higher rewards faster than baselines.

Limitations and Future Work

  • Transient Instability: Like many O2O methods, HiP3 still experiences a brief performance dip when first switching from offline to online. Mitigating this "initial shock" remains an open challenge.
  • Model Bias: The optimism and pessimism proxies rely on a learned world model. If the model is severely biased, the Pareto front itself might be misleading.
  • Scalability: While it works well on MuJoCo, covering the Pareto front in extremely high-dimensional or complex environments might require a much larger (and computationally expensive) policy pool.

For those interested in the implementation, the code is available at https://github.com/GuochenZhou/HiP3.

Summary updated

Abstract

Model-based offline-to-online reinforcement learning (RL) enables sample-efficient adaptation by leveraging offline pre-training for online fine-tuning. However, the distribution shifts between offline and online stages often hinder fine-tuning performance. Many existing methods encode the optimism-pessimism trade-off through a single-objective formulation. Selecting its trade-off coefficient through online fine-tuning can require costly bi-level optimization. We identify the balance established during offline training as a key challenge: in our motivating experiments, optimistic policies adapt better to novel tasks, whereas pessimistic policies perform better on the same online tasks. To address this challenge, we propose a bi-objective formulation that captures this trade-off, yielding a Pareto policy pool during offline training. This policy pool enables adaptive selection during online fine-tuning across diverse online tasks. To generate the pool, we introduce Multi-Objective Soft Actor-critIC (MOSAIC), which solves bi-objective problems and constructs diverse Pareto policies. After offline training, a contextual bandit algorithm hierarchically selects the most suitable policy for fine-tuning at each online interaction step. Empirically, our pipeline, Hierarchical Pareto Policy Pool (HiP3), achieves the strongest aggregate performance across both same and novel online tasks and maintains its advantage under stronger distribution shifts. Comprehensive ablations further clarify the roles of the policy pool, MOSAIC-based pool construction, and the online selection mechanism.
BibTeX
@misc{zhou2026towards,
  author = {Guochen Zhou and Yijun Yang and Qiqi Duan and Qing Su and Weiming Ou and Li Shen and Shihao Ji and Yuhui Shi and Zhisong Zhang},
  title = {{Towards Optimism-Pessimism Trade-off in Model-based Offline-to-Online Reinforcement Learning}},
  year = {2026},
  eprint = {2611.22172},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.22172},
}
Figure