Behavioral Cloning via Search in Embedded Demonstration Dataset

Federico Malato, Florian Leopold, Ville Hautamäki, Andrew MelnikarXiv 2023 · DOI · arXiv

Introduction

Imitation Learning (IL) and Reinforcement Learning (RL) have traditionally relied on training complex neural network policies to map states directly to actions. However, in complex, multi-modal environments like Minecraft, this approach faces significant hurdles, including high computational costs for training foundation models like Video Pre-Training (VPT), brittleness in novel situations, and a lack of adaptability to new tasks without extensive fine-tuning.

This paper proposes a paradigm shift: instead of learning a policy through backpropagation, we can treat control as a retrieval problem. The authors introduce Search-based Behavioral Cloning (S-BC), a method that leverages a pre-trained latent space to index expert demonstrations and "copy-paste" their behavior in real-time.

The S-BC Framework

The core of S-BC is the transformation of a control problem into a k-Nearest Neighbors search over expert trajectories. The process is divided into an offline indexing phase and an online execution phase.

Overview of the S-BC pipeline: (A) Offline creation of the Situations Dataset using a frozen VPT encoder; (B) Online inference loop retrieving actions based on L1 distance in latent space.

As shown in the architecture above, S-BC utilizes a frozen VPT model as an encoder to transform raw video frames into high-dimensional latent embeddings. A database of expert demonstrations is processed to create a "searchable" library of situations. During evaluation, the agent encodes its current observation and identifies the most similar situation in the expert database using an L1 distance metric.

Search and Retrieval Mechanism

The effectiveness of S-BC relies on the semantic richness of the latent space. The authors demonstrate that the VPT encoder naturally clusters similar environmental states, allowing for reliable retrieval.

Visualization of the search mechanism showing the L1 distance between the current state and the retrieved expert state over time, with corresponding frame comparisons.

Once a match is found, the agent follows the expert’s subsequent actions. To maintain coherence, the agent employs two triggers for a new search:

  1. Divergence-triggered search: Occurs when the agent’s current state drifts too far in the latent space from the expert’s path.
  2. Time-triggered search: Occurs after a fixed number of steps to ensure the agent remains oriented toward the goal.

This trajectory-following approach avoids the "action averaging" problem common in standard Behavioral Cloning, where a model might fail by trying to average distinct expert responses to the same state.

Key Results and Efficiency

S-BC demonstrates remarkable performance and efficiency compared to traditional learning-based methods:

  • Performance: In the NeurIPS BASALT 2022 "FindCave" challenge, S-BC achieved a 40% success rate, outperforming fine-tuned VPT models (15-30%) and ranking 2nd overall.
  • Computational Efficiency: "Training" S-BC for a new task involves a single forward pass to index the data, taking roughly 30 minutes on a single Tesla T4 GPU. In contrast, fine-tuning or training GAIL (Generative Adversarial Imitation Learning) requires 3 to 6 hours.
  • Task Adaptability: Switching tasks is as simple as swapping the demonstration dataset; no weights are updated, enabling near-instantaneous task adaptation.

Conclusion

Search-based Behavioral Cloning offers a compelling alternative to traditional policy training in complex environments. By leveraging the structured latent spaces of foundation models, S-BC achieves state-of-the-art results with minimal computational overhead. While sensitive to the quality of the demonstration dataset, its ability to provide decisive, human-like behavior makes it a powerful tool for scalable imitation learning.

Summary updated

Abstract

Behavioural cloning uses a dataset of demonstrations to learn a behavioural policy. To overcome various learning and policy adaptation problems, we propose to use latent space to index a demonstration dataset, instantly access similar relevant experiences, and copy behavior from these situations. Actions from a selected similar situation can be performed by the agent until representations of the agent's current situation and the selected experience diverge in the latent space. Thus, we formulate our control problem as a search problem over a dataset of experts' demonstrations. We test our approach on BASALT MineRL-dataset in the latent representation of a Video PreTraining model. We compare our model to state-of-the-art Minecraft agents. Our approach can effectively recover meaningful demonstrations and show human-like behavior of an agent in the Minecraft environment in a wide variety of scenarios. Experimental results reveal that performance of our search-based approach is comparable to trained models, while allowing zero-shot task adaptation by changing the demonstration examples.
BibTeX
@misc{malato2023behavioral,
  author = {Federico Malato and Florian Leopold and Ville Hautamäki and Andrew Melnik},
  title = {{Behavioral Cloning via Search in Embedded Demonstration Dataset}},
  year = {2023},
  eprint = {2306.09082},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2306.09082},
}
Figure