Chain of Thought Imitation with Procedure Cloning

Mengjiao Yang, Dale Schuurmans, Pieter Abbeel, Ofir Nachum

Abstract

Imitation learning typically frames the problem of learning from expert demonstrations as a supervised mapping from observations to actions. However, this approach often ignores the underlying computational procedures—such as search, planning, or optimization—that experts use to derive those actions. We introduce Procedure Cloning (PC), a method that applies supervised sequence prediction to imitate the series of intermediate computations performed by an expert. By learning not just what action to take, but the "chain of thought" leading to it, PC enables agents to generalize to unseen environment configurations where standard Behavioral Cloning (BC) fails. Our evaluation across navigation, robotic manipulation, and game-playing environments demonstrates that PC significantly outperforms traditional imitation learning by leveraging the rich, procedural information inherent in expert behavior.

Introduction

Behavioral Cloning (BC) has long been the workhorse of imitation learning due to its simplicity. By treating the expert as a black box and fitting a policy to state-action pairs, BC can be applied to almost any domain. However, this simplicity comes at a cost: brittleness. When an agent encounters a state slightly outside the training distribution, it often fails because it has only learned a surface-level mapping rather than the underlying logic of the task.

In many real-world and simulated settings, expert demonstrations are generated by algorithms like A search, Monte Carlo Tree Search (MCTS), or scripted controllers. These algorithms do not just produce an action; they follow a systematic procedure involving intermediate steps, backtracking, and sub-goal evaluation. Procedure Cloning (PC) posits that these intermediate steps are not just "noise" to be discarded, but essential data that explains why* an action was chosen. By supervising the agent on these internal traces, we can train models that "think" before they act.

Methodology: From Actions to Procedures

Standard imitation learning seeks to model the conditional distribution $p(a | s)$. In contrast, Procedure Cloning models the joint distribution of the action and the sequence of intermediate "procedure observations" $\mathbf{x} = (x_1, x_2, ..., x_n)$ that the expert generated while deciding on that action.

Procedure Observations

Procedure observations are snapshots of the expert's internal state. For a search-based expert, these might be the coordinates of nodes currently being expanded in a search tree. For a robotic controller, they might be the target coordinates of a series of waypoints. Crucially, while the expert may use privileged information or tools (like a simulator) to generate these steps, the PC agent only needs to learn to predict the sequence of steps from the initial observation.

Sequence Modeling and Inference

PC treats imitation as a sequence prediction problem, making it highly compatible with modern Transformer architectures. During training, the model is supervised to predict the entire trace of the expert's computation. At inference time, the agent does not have access to the expert's internal tools. Instead, it uses its learned model to autoregressively generate its own internal chain of thought. It "imagines" the search process or the sub-goals internally before committing to a final physical action.

Experimental Results

We evaluated Procedure Cloning across four diverse domains to test its ability to generalize and handle complex tasks.

In a synthetic 32x32 maze environment where the expert uses Breadth-First Search (BFS), standard BC and its variants (including those with auxiliary tasks) achieved 0% success on new, unseen layouts. In contrast, Procedure Cloning achieved 100% success. By predicting the BFS expansion steps, the PC agent effectively learned the logic of pathfinding rather than just memorizing specific routes.

Robotic Manipulation

In the Bimanual Sweep task, two robotic arms must coordinate to sweep particles into bowls. This task requires high precision and handling of privileged information. PC achieved a success metric of 83.9%, significantly outperforming the previous state-of-the-art, Implicit BC, which reached 78.2%. This suggests that imitating the expert's sub-goal generation provides a more robust policy for complex physical coordination.

Strategic Game Playing

Using the MinAtar environment with an MCTS expert, we found that PC agents were remarkably robust. When game difficulty was increased or stochasticity ("sticky actions") was introduced at test time, PC agents maintained high performance. In some instances, the PC agent even outperformed the MCTS expert it was trained on, likely because the sequence modeling process smoothed out the expert's search noise into a more robust representation of game logic.

Discussion and Future Work

Procedure Cloning represents a shift in how we view imitation learning. By treating the expert's internal computations as a "chain of thought," we can leverage the density of supervision provided by modern sequence modeling.

However, PC does introduce new challenges. Generating a full sequence of internal steps for every action is computationally more expensive than a single policy forward pass, which may limit its use in high-frequency real-time control. Additionally, the success of PC depends on the practitioner's ability to identify and record meaningful intermediate steps from the expert. Future work will explore how to automatically discover these procedural traces or use natural language explanations as the "procedure" to further bridge the gap between human reasoning and robotic execution.

Summary updated

Abstract

Imitation learning aims to extract high-performance policies from logged demonstrations of expert behavior. It is common to frame imitation learning as a supervised learning problem in which one fits a function approximator to the input-output mapping exhibited by the logged demonstrations (input observations to output actions). While the framing of imitation learning as a supervised input-output learning problem allows for applicability in a wide variety of settings, it is also an overly simplistic view of the problem in situations where the expert demonstrations provide much richer insight into expert behavior. For example, applications such as path navigation, robot manipulation, and strategy games acquire expert demonstrations via planning, search, or some other multi-step algorithm, revealing not just the output action to be imitated but also the procedure for how to determine this action. While these intermediate computations may use tools not available to the agent during inference (e.g., environment simulators), they are nevertheless informative as a way to explain an expert's mapping of state to actions. To properly leverage expert procedure information without relying on the privileged tools the expert may have used to perform the procedure, we propose procedure cloning, which applies supervised sequence prediction to imitate the series of expert computations. This way, procedure cloning learns not only what to do (i.e., the output action), but how and why to do it (i.e., the procedure). Through empirical analysis on navigation, simulated robotic manipulation, and game-playing environments, we show that imitating the intermediate computations of an expert's behavior enables procedure cloning to learn policies exhibiting significant generalization to unseen environment configurations, including those configurations for which running the expert's procedure directly is infeasible.
BibTeX
@inproceedings{yang2022chain,
  author = {Mengjiao Yang and Dale Schuurmans and Pieter Abbeel and Ofir Nachum},
  title = {{Chain of Thought Imitation with Procedure Cloning}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2022},
  url = {https://openreview.net/forum?id=3byEXeqlamX},
}
Figure