Filtered Behavior Cloning for Policy Optimization

32 concepts29 papers1 proposal

Reinforcement learning (RL) and imitation learning (IL) have long operated under a fundamental trade-off. Traditional RL algorithms can discover optimal policies from scratch but are notoriously unstable, sample-inefficient, and highly sensitive to hyperparameter tuning. Conversely, Behavior Cloning (BC) is exceptionally stable and fast because it reduces policy learning to simple supervised learning. However, BC possesses a fatal flaw: it blindly mimics its training data, meaning it replicates every mistake, sub-optimal detour, and failure present in the dataset.

Filtered Behavior Cloning (FBC) resolves this tension through a simple but powerful insight: instead of forcing an agent to learn from an entire uncurated dataset or a messy history of exploration, we can apply a strict performance filter to isolate only the highest-quality trajectories or transitions. Once the noise and failures are pruned away, a standard supervised model can be trained exclusively on these "winning" strategies. This approach combines the optimization power of RL with the training stability of supervised learning.

The Core Intuition: Pruning the Path to Success

Why filter data instead of letting an algorithm learn from both successes and failures? In complex environments with sparse rewards, the action space is vast, and there are infinitely more ways to fail than to succeed. Introducing negative rewards or penalizing bad trajectories often fails to provide a clear gradient toward the correct behavior; it merely tells the agent what not to do.

By filtering the data to retain only optimal behaviors, the learning task is dramatically simplified. As Omori demonstrates, simply training a standard multilayer perceptron on the top 10% of trajectories in an offline dataset can consistently outperform complex sequence-modeling architectures like the Decision Transformer, while requiring a fraction of the training time and parameters.

This selective curation prevents the agent from adopting the inefficient or meandering behaviors common in uncurated datasets. For example, Moon et al. show that filtering training data to include only trajectories that meet strict efficiency thresholds allows path-planning robots to develop robust, direct navigation strategies rather than replicating suboptimal paths.

Coarse vs. Fine-Grained Filtering Mechanisms

Filtering can occur at different levels of granularity, depending on whether the evaluation metric is applied to entire episodes, sub-trajectories, or individual state-action transitions.

Episode-Level Filtering

The most straightforward approach filters whole trajectories based on cumulative metrics. Tuyls et al. employ a selection mechanism that favors trajectories with both high cumulative scores and minimal path lengths. This dual constraint ensures that the agent replicates efficient progress rather than simply repeating any sequence that happened to stumble upon a reward. In batch reinforcement learning, Chen introduces Best-Action Imitation Learning (BAIL), which fits an "upper envelope" to a static dataset to isolate and clone only the highest-performing samples, completely bypassing the need for complex policy constraints.

Sub-Trajectory and Transition-Level Filtering

Often, an overall episode might end in failure, yet contain highly instructive sequences. To capture these moments, Hao et al. extract and store high-performing sub-trajectories, allowing independent agents to coordinate and learn specific useful maneuvers even if the overall mission outcome was unsuccessful.

At the finest granularity, value functions or reward models can act as dynamic, step-by-step filters. Chen et al. utilize a learned reward model to score demonstrations, automatically down-weighting or filtering out stagnant or regressive actions so the policy avoids mimicking noise.

Similarly, Kim et al. implement a value-based "Q-filter" that evaluates a demonstration action against the agent's current policy predictions; if the agent's self-generated action is predicted to yield a higher reward, the suboptimal demonstration is ignored.

Taking this further, Zhang et al. use the current value function to filter self-imitation data, applying a behavior cloning loss only to specific actions that are predicted to actively reduce the distance to the goal, preventing the agent from copying its own meandering exploration history.

Uncurated Trajectories ───► [ Performance Filter ] ───► High-Quality Subsets ───► Supervised BC ───► Optimized Policy
(Exploration/Noisy Data)     - Return Thresholds          (Optimal Actions Only)
                             - Value Functions
                             - Success Evaluators

Self-Improvement via Online Rollout Filtering

One of the most powerful applications of this paradigm is autonomous self-improvement, particularly for Large Language Model (LLM) and vision-language agents. Instead of relying on scarce human expert demonstrations, an agent can explore an environment, generate its own rollouts, and use an automated evaluator to filter its own history.

Shen et al. and Shen et al. explore this through success-filtered behavioral cloning, where an interactive agent scales its test-time environment steps and fine-tunes its policy exclusively on its own successful rollouts. This ensures that the learning signal is derived from high-quality examples that are natively within the model's current capability distribution. In software engineering tasks, Wang et al. use a similar sampling phase to filter out unsuccessful code-generation attempts, reinforcing only the successful paths to bootstrap agent performance.

To make this process fully autonomous, the evaluation itself can be offloaded to foundation models. Zhou et al. leverage the "asymmetric gap"—the fact that large models are often significantly better at evaluating a task than executing it—to build a Proposer-Agent-Evaluator framework. The automated evaluator filters the agent's wild exploration, feeding only the verified successful trajectories back into the behavior cloning loop. Even when extracting underlying logic, filtering proves vital: Meli et al. collect belief-action pairs exclusively from successful execution runs to serve as clean training examples for inducing human-readable logic rules from complex agents.

High-Fidelity Distillation and Cross-Agent Guidance

Filtering also acts as a crucial safeguard when transferring knowledge between different models, peers, or tasks, ensuring that errors or sub-optimal strategies are not propagated to the student.

When distilling massive cloud-hosted models into compact, on-device models, Ravichandran et al. filter synthetic teacher trajectories to include only successful task completions, discarding instances where the teacher model fails or times out. This curation process ensures the student model is trained exclusively on winning strategies. Similarly, in multi-turn strategic reasoning games, Zhang et al. demonstrate that open-source models can be dramatically improved when fine-tuned exclusively on successful interaction trajectories generated by a more capable teacher model.

This principle extends naturally to multi-agent and multi-task settings where agents must learn from external sources:

  • Learning from Peers: In the Co-Imitation Learning without Expert Demonstration framework, peer experiences are filtered through utility estimation. This ensures that agents only adopt behaviors from their peers that exceed their own current value expectations, preventing the propagation of sub-optimal actions.
  • Learning from Other Tasks: For multi-task reinforcement learning, He et al. utilize a gating mechanism that filters potential cross-task guidance by comparing the value function of the current task against the estimated quality of a suggested policy, ensuring external behaviors are only adopted when they guarantee an improvement.

Why It Works: The Intuitive and Theoretical Edge

Why does filtering out bad data work better than traditional RL methods that explicitly penalize bad data?

First, it bypasses the optimization pitfalls of value-based RL. As Zhou et al. demonstrate, traditional Q-learning methods often fail when paired with deep neural networks due to the strict requirement of Bellman completeness, which causes errors to compound over time. By using a dynamics model to generate synthetic trajectories and filtering them to retain only those surpassing the maximum observed dataset return, the authors achieve "trajectory stitching" via pure supervised learning, bypassing Bellman constraints entirely.

Second, positive-only filtering simplifies the gradient path. As argued by Chen et al., updating a policy using only successful interaction sequences can be significantly more effective than incorporating negative rewards. In large action spaces, negative signals fail to provide a clear gradient toward the correct behavior. Filtered behavior cloning provides a direct, unambiguous target.

Finally, it redefines how we leverage imperfect data. Instead of discarding noisy datasets, methods like the one proposed in How to Leverage Diverse Demonstrations in Offline Imitation Learning filter transitions based on where they lead. By selecting actions from imperfect data that successfully transition the agent back into an expert state distribution, the agent learns how to recover from errors without ever mimicking the suboptimal behaviors that caused them.

Key Foundational Work

Independent Generative Adversarial Self-Imitation Learning in Cooperative Multiagent Systems (2019) — contributed 1 concept elements to this direction.
Développement autonome des comportements de base d'un agent (2005) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_filtered_behavior_cloning_f11b1ebb,
  title = {filtered behavior cloning},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/filtered-behavior-cloning/rdr_f11b1ebb3efd495b9a300d0543f01e78},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-22},
}

This exact version (version 1, frozen text):

@misc{lacuna_filtered_behavior_cloning_f11b1ebb_v1,
  title = {filtered behavior cloning},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/filtered-behavior-cloning/rdr_f11b1ebb3efd495b9a300d0543f01e78/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-22},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake