Posterior Behavioral Cloning: Pretraining BC Policies for Efficient RL Finetuning

Andrew Wagenmaker, Perry Dong, Raymond Tsao, Chelsea Finn, Sergey Levine

TL;DR

Standard behavioral cloning (BC) pretraining often overfits to observed demonstrations, causing downstream reinforcement learning (RL) finetuning to fail due to a lack of action coverage. To solve this, the authors introduce Posterior Behavioral Cloning (POSTBC), a purely supervised method that models the posterior distribution of the demonstrator's behavior. By injecting uncertainty-aware entropy into the pretrained policy, POSTBC dramatically improves RL finetuning efficiency without sacrificing initial zero-shot performance.

What Problem Does This Paper Address?

The standard recipe for building capable robots or language models is simple: pretrain a policy on human demonstrations via Behavioral Cloning (BC), then finetune it using Reinforcement Learning (RL) to maximize performance. While much research focuses on designing better RL finetuning algorithms, very little attention is paid to the quality of the initialization provided by the pretrained policy.

The authors identify a fundamental flaw in standard BC pretraining: it acts as a Maximum A Posteriori (MAP) estimate, exactly mimicking observed actions but assigning near-zero probability to unseen actions. In regions with low data density, standard BC overcommits to a narrow set of behaviors. When you try to finetune this policy with online RL, the robot cannot explore alternative, potentially better actions, leading to a lack of informative reward signals and stagnant performance. Simply adding uniform exploration noise ($\sigma$-BC) doesn't work either, as it degrades the policy's initial performance and is highly sample-inefficient.

Methods

The authors propose Posterior Behavioral Cloning (POSTBC). Instead of treating pretraining as a point-estimation problem, POSTBC models the full posterior distribution of the demonstrator's behavior given the dataset.

The practical implementation consists of two scalable, supervised learning steps:

  1. Posterior Variance Approximation via Ensembled Prediction: They train an ensemble of $K$ simple MLP predictors on bootstrapped versions of the demonstration dataset. The variance across these ensemble predictions serves as a proxy for the policy's epistemic uncertainty at any given state.
  2. Posterior Behavioral Cloning: They train the final generative policy (parameterized as a diffusion model) on the demonstration dataset. Crucially, during training, they perturb the target actions by adding Gaussian noise scaled by the calculated posterior covariance matrix and a hyperparameter $\alpha$.

This approach ensures that in states where data is sparse (high uncertainty), the policy samples from a high-entropy distribution to encourage exploration. In states with abundant data (low uncertainty), it reduces to standard BC, precisely mimicking the demonstrator.

Datasets

The method was evaluated across both simulated benchmarks and real-world tasks:

  • Robomimic: Single-task datasets using state-based observations for the Lift, Can, and Square manipulation tasks.
  • Libero (Libero 90 suite): A multi-task benchmark where a single policy was pretrained on 16 distinct language-conditioned tasks across three kitchen scenes using image-based observations.
  • Real-World WidowX Robot: A 6-DoF robot arm evaluated on two physical tasks: Put corn in pot (10 human teleoperation demonstrations) and a zero-shot generalization task, Pick up banana.

Key Insights and Ideas

Overview of standard BC vs. POSTBC pretraining and subsequent RL finetuning.
  • Uncertainty-Aware Entropy is Free: Adding entropy to a policy in states where you don't know the correct action doesn't hurt performance relative to BC. Since the BC policy might be wrong anyway in low-density regions, expanding the action distribution to cover all plausible demonstrator actions is mathematically optimal.
  • The "Test-Time" Sampling Utility: Through ablation studies, the authors decoupled the exploration policy (data collection) from the test-time policy. They discovered that POSTBC's primary strength lies in providing a wider, cleaner distribution of actions at test time, allowing finetuning methods like Best-of-N sampling to easily filter and select the best possible action.
  • No RL Required for Pretraining: Unlike offline RL methods that require reward labels and complex value-function tuning, POSTBC achieves exploration-ready policies using only standard, stable supervised learning pipelines.

Key Results

  • Preserved Pretrained Performance: Across all Robomimic and Libero tasks, POSTBC matched or slightly exceeded the zero-shot success rates of standard BC, whereas uniform noise ($\sigma$-BC) consistently degraded performance.
  • Dramatically Better RL Finetuning: When combined with finetuning algorithms like DSRL and DPPO, POSTBC achieved massive sample-efficiency gains. On the challenging Robomimic Square task, POSTBC required roughly 2x fewer samples to reach a 75% success rate compared to standard BC.
  • Real-World Validation: On the physical WidowX robot, combining POSTBC with Best-of-N sampling boosted the success rate of the Put corn in pot task from 35% (pretrained) to 65% (finetuned)—a 30% absolute improvement, compared to just a 10% improvement when using standard BC.

Limitations and Future Work

  • Necessity vs. Sufficiency: While the authors prove that demonstrator action coverage is a necessary condition for effective RL finetuning, it is not a sufficient one. It guarantees the right actions can be sampled, but does not guarantee downstream sample complexity bounds.
  • Supervised Learning Constraint: The framework is strictly bounded by supervised learning. The authors note that exploring whether offline RL could provide an even better initialization remains an open question.
  • Extension to Language Domains: POSTBC was heavily validated on continuous robotic control tasks. A promising future direction is adapting this uncertainty-aware pretraining to discrete next-token prediction in Large Language Models (LLMs) to improve RLHF (Reinforcement Learning from Human Feedback) efficiency.

Summary updated

Abstract

Standard practice across domains from robotics to language is to first pretrain a policy on a large-scale demonstration dataset, and then finetune this policy, typically with reinforcement learning (RL), in order to improve performance on deployment domains. This finetuning step has proved critical in achieving human or super-human performance, yet while much attention has been given to developing more effective finetuning algorithms, little attention has been given to ensuring the pretrained policy is an effective initialization for RL finetuning. In this work we seek to understand how the pretrained policy affects finetuning performance, and how to pretrain policies in order to ensure they are effective initializations for finetuning. We first show theoretically that standard behavioral cloning (BC) -- which trains a policy to directly match the actions played by the demonstrator -- can fail to ensure coverage over the demonstrator's actions, a minimal condition necessary for effective RL finetuning. We then show that if, instead of exactly fitting the observed demonstrations, we train a policy to model the posterior distribution of the demonstrator's behavior given the demonstration dataset, we do obtain a policy that ensures coverage over the demonstrator's actions, enabling more effective finetuning. Furthermore, this policy -- which we refer to as the posterior behavioral cloning (PostBC) policy -- achieves this while ensuring pretrained performance is no worse than that of the BC policy. We then show that PostBC is practically implementable with modern generative models in robotic control domains -- relying only on standard supervised learning -- and leads to significantly improved RL finetuning performance on both realistic robotic control benchmarks and real-world robotic manipulation tasks, as compared to standard behavioral cloning.
BibTeX
@inproceedings{wagenmaker2026posterior,
  author = {Andrew Wagenmaker and Perry Dong and Raymond Tsao and Chelsea Finn and Sergey Levine},
  title = {{Posterior Behavioral Cloning: Pretraining BC Policies for Efficient RL Finetuning}},
  booktitle = {International Conference on Machine Learning},
  year = {2026},
  url = {https://openreview.net/forum?id=HJ5F3R9Tmd},
}
Figure