TL;DR
In offline-to-online reinforcement learning, discarding the offline dataset during fine-tuning usually leads to catastrophic performance collapse. This paper reveals that the culprit is continued behavioral cloning on limited online data, which triggers a "representational collapse"; simply freezing the expressive behavioral prior while allowing the critic to learn enables robust, sustained improvement without needing to store the original data.
What Problem Does This Paper Address?
The standard pipeline for modern RL involves offline pretraining (learning from a fixed dataset) followed by online fine-tuning (improving through environment interaction). Most state-of-the-art methods assume the offline dataset remains available forever to act as a "buffer" that prevents the agent from drifting too far from known good behaviors.
However, in the real world, keeping that data is often impossible due to storage costs, privacy regulations, or licensing restrictions. This is the "no-retention" setting. When the offline data is deleted, agents typically experience a rapid loss of capabilities. While previous research blamed this on the "critic" (the value function) failing to recalibrate to new data, this paper identifies a critical, complementary failure in the "actor" (the policy).
Methods
The authors propose a deceptively simple design principle: Keep learning which actions are valuable, but stop relearning which actions are possible.
They introduce BC-OFF, a strategy where the behavioral prior—the model that understands the distribution of valid actions—is frozen immediately after offline pretraining. Meanwhile, the critic and the decision-making components continue to update using only new online experience.
Key to this approach is the use of expressive policies, such as Flow-based or Diffusion models. These models don't just learn one "average" action; they learn a complex manifold of many possible valid actions. The authors test this across three major algorithmic frameworks:
- IFQL (Implicit Flow Q-Learning): The frozen flow model proposes a "menu" of diverse actions, and the updated critic selects the best one.
- FQL (Flow Q-Learning): A one-step actor is distilled toward the frozen flow prior while being optimized for value.
- QC (Action-Chunked RL): The agent preserves a prior over sequences of actions (chunks) rather than single steps.
Datasets
The study is remarkably thorough, evaluating the approach on:
- OGBench: A suite of 7 domains (locomotion and manipulation) encompassing 35 distinct tasks. This includes challenging "noisy" and "explore" variants to test performance on low-quality data.
- Robomimic: Specifically the "Multi-Human" (MH) datasets for the Can and Lift tasks, which feature realistic suboptimality and multimodality from six different human operators.
Key Insights and Ideas
- The Behavioral Feedback Loop: When you continue Behavioral Cloning (BC) on a small online buffer, the model starts imitating its own recent, narrow, and often suboptimal experiences. This "cannibalizes" the broad knowledge learned offline.
- Representational Collapse: The authors found that continued BC causes the internal features of the actor network to lose dimensionality (rank collapse). This destroys action diversity, leaving the agent trapped in repetitive failure modes.
- Data Starvation of the Critic: Because the actor's behavior collapses into "jitter," it stops exploring the environment effectively. The critic, now fed a "narrow, repetitive diet of transitions," also suffers representational collapse because it no longer sees enough diversity to distinguish good states from bad ones.
- Expressivity is Non-Negotiable: Simply freezing a policy isn't enough. If the prior is a simple Gaussian (unimodal), it fails. The prior must be expressive and multimodal to provide the critic with a rich enough "search space" of physically valid actions to choose from.
Key Results
- Universal Improvement: Freezing the prior (BC-OFF) consistently improved performance across all 35 OGBench tasks compared to the standard BC-ON approach.
- Rescuing Performance: In many cases, BC-OFF allowed the agent to match the performance of "retention" baselines that keep the entire offline dataset (see Figure 1).
- Sensitivity to Weighting: The authors showed that even aggressively lowering the BC loss weight (e.g., to 0.1x) cannot prevent collapse. Only completely removing the objective (BC-OFF) works, as shown in Figure 2.
- Robustness to Data Quality: Even when pretrained on low-quality, noisy data, a frozen expressive prior acts as a "valid action manifold," constraining the agent to physically sensible movements while the critic learns to pick the best ones.
Limitations and Future Work
- Environment Shifts: A strictly frozen prior assumes the underlying physics and valid action space don't change. If an agent moves to an environment with significantly different dynamics, a hard-frozen prior might become a liability.
- Generative Replay: The authors suggest that instead of hard freezing, future work could use the expressive prior to generate synthetic "offline" data, allowing for more flexible but safe updates.
- Scaling to VLA: A promising direction is applying these "freeze" principles to massive Vision-Language-Action (VLA) models, where retaining the original training data is practically impossible.
Summary updated