Freeze, Don't Clone: Fine-Tuning with Expressive Policies Without Retaining Offline Data

Sami Nur Islam, Johan Obando-Ceron, Doina Precup, Aaron Courville, Pablo Samuel CastroarXiv 2026 · arXiv

TL;DR

In offline-to-online reinforcement learning, discarding the offline dataset during fine-tuning usually leads to catastrophic performance collapse. This paper reveals that the culprit is continued behavioral cloning on limited online data, which triggers a "representational collapse"; simply freezing the expressive behavioral prior while allowing the critic to learn enables robust, sustained improvement without needing to store the original data.

What Problem Does This Paper Address?

The standard pipeline for modern RL involves offline pretraining (learning from a fixed dataset) followed by online fine-tuning (improving through environment interaction). Most state-of-the-art methods assume the offline dataset remains available forever to act as a "buffer" that prevents the agent from drifting too far from known good behaviors.

However, in the real world, keeping that data is often impossible due to storage costs, privacy regulations, or licensing restrictions. This is the "no-retention" setting. When the offline data is deleted, agents typically experience a rapid loss of capabilities. While previous research blamed this on the "critic" (the value function) failing to recalibrate to new data, this paper identifies a critical, complementary failure in the "actor" (the policy).

Methods

The authors propose a deceptively simple design principle: Keep learning which actions are valuable, but stop relearning which actions are possible.

They introduce BC-OFF, a strategy where the behavioral prior—the model that understands the distribution of valid actions—is frozen immediately after offline pretraining. Meanwhile, the critic and the decision-making components continue to update using only new online experience.

Key to this approach is the use of expressive policies, such as Flow-based or Diffusion models. These models don't just learn one "average" action; they learn a complex manifold of many possible valid actions. The authors test this across three major algorithmic frameworks:

  • IFQL (Implicit Flow Q-Learning): The frozen flow model proposes a "menu" of diverse actions, and the updated critic selects the best one.
  • FQL (Flow Q-Learning): A one-step actor is distilled toward the frozen flow prior while being optimized for value.
  • QC (Action-Chunked RL): The agent preserves a prior over sequences of actions (chunks) rather than single steps.

Datasets

The study is remarkably thorough, evaluating the approach on:

  • OGBench: A suite of 7 domains (locomotion and manipulation) encompassing 35 distinct tasks. This includes challenging "noisy" and "explore" variants to test performance on low-quality data.
  • Robomimic: Specifically the "Multi-Human" (MH) datasets for the Can and Lift tasks, which feature realistic suboptimality and multimodality from six different human operators.

Key Insights and Ideas

  • The Behavioral Feedback Loop: When you continue Behavioral Cloning (BC) on a small online buffer, the model starts imitating its own recent, narrow, and often suboptimal experiences. This "cannibalizes" the broad knowledge learned offline.
  • Representational Collapse: The authors found that continued BC causes the internal features of the actor network to lose dimensionality (rank collapse). This destroys action diversity, leaving the agent trapped in repetitive failure modes.
  • Data Starvation of the Critic: Because the actor's behavior collapses into "jitter," it stops exploring the environment effectively. The critic, now fed a "narrow, repetitive diet of transitions," also suffers representational collapse because it no longer sees enough diversity to distinguish good states from bad ones.
  • Expressivity is Non-Negotiable: Simply freezing a policy isn't enough. If the prior is a simple Gaussian (unimodal), it fails. The prior must be expressive and multimodal to provide the critic with a rich enough "search space" of physically valid actions to choose from.
Figure 1: Stopping BC enables no-retention fine-tuning. Left: After offline pretraining, we freeze the behavioral prior and continue updating the critic using online data. Right: Aggregate performance across seven OGBench tasks after 1M offline gradient steps (gray) and 1M online environment steps (white). Stopping BC consistently improves methods with expressive policies.

Key Results

  • Universal Improvement: Freezing the prior (BC-OFF) consistently improved performance across all 35 OGBench tasks compared to the standard BC-ON approach.
  • Rescuing Performance: In many cases, BC-OFF allowed the agent to match the performance of "retention" baselines that keep the entire offline dataset (see Figure 1).
  • Sensitivity to Weighting: The authors showed that even aggressively lowering the BC loss weight (e.g., to 0.1x) cannot prevent collapse. Only completely removing the objective (BC-OFF) works, as shown in Figure 2.
  • Robustness to Data Quality: Even when pretrained on low-quality, noisy data, a frozen expressive prior acts as a "valid action manifold," constraining the agent to physically sensible movements while the critic learns to pick the best ones.
Figure 2: Sensitivity to the BC loss weight. Online fine-tuning performance across vary- ing BC loss weights (0.1× to 10× default) without offline data retention. Scaling the BC weight up or down (yellow to dark brown) fails to prevent collapse in both IFQL and FQL. Only removing the BC loss entirely (BC-OFF, green) avoids collapse and enables sustained online learning.

Limitations and Future Work

  • Environment Shifts: A strictly frozen prior assumes the underlying physics and valid action space don't change. If an agent moves to an environment with significantly different dynamics, a hard-frozen prior might become a liability.
  • Generative Replay: The authors suggest that instead of hard freezing, future work could use the expressive prior to generate synthetic "offline" data, allowing for more flexible but safe updates.
  • Scaling to VLA: A promising direction is applying these "freeze" principles to massive Vision-Language-Action (VLA) models, where retaining the original training data is practically impossible.

Summary updated

Abstract

Offline-to-online reinforcement learning uses offline pretraining to accelerate subsequent online learning. While many methods assume the offline dataset remains available throughout fine-tuning, there are many settings where this is not the case. Prior work attributes failures in this setting primarily to critic recalibration under distribution shift. We identify a complementary actor-side failure: once the offline data disappears, continued behavioral cloning (BC) no longer preserves the offline behavior distribution and instead refits the behavioral model to the agent's limited, and nonstationary, online experience. We demonstrate that stopping online behavioral fitting while preserving the pretrained expressive prior substantially improves no-retention fine-tuning across implicit flow Q-learning (IFQL), flow Q-learning (FQL), and action-chunked RL (QC) on seven OGBench domains. We trace this failure to a behavioral feedback loop: continuing online behavioral cloning rapidly erases the pretrained prior, inducing representational collapse and action diversity loss that starves the critic of useful exploration. Freezing the prior breaks this cycle, preserving offline coverage while allowing value learning to drive continuous improvement.
BibTeX
@misc{islam2026freeze,
  author = {Sami Nur Islam and Johan Obando-Ceron and Doina Precup and Aaron Courville and Pablo Samuel Castro},
  title = {{Freeze, Don't Clone: Fine-Tuning with Expressive Policies Without Retaining Offline Data}},
  year = {2026},
  eprint = {2610.36247},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.36247},
}
Figure