Abstract
Imitation learning typically frames the problem of learning from expert demonstrations as a supervised mapping from observations to actions. However, this approach often ignores the underlying computational procedures—such as search, planning, or optimization—that experts use to derive those actions. We introduce Procedure Cloning (PC), a method that applies supervised sequence prediction to imitate the series of intermediate computations performed by an expert. By learning not just what action to take, but the "chain of thought" leading to it, PC enables agents to generalize to unseen environment configurations where standard Behavioral Cloning (BC) fails. Our evaluation across navigation, robotic manipulation, and game-playing environments demonstrates that PC significantly outperforms traditional imitation learning by leveraging the rich, procedural information inherent in expert behavior.
Introduction
Behavioral Cloning (BC) has long been the workhorse of imitation learning due to its simplicity. By treating the expert as a black box and fitting a policy to state-action pairs, BC can be applied to almost any domain. However, this simplicity comes at a cost: brittleness. When an agent encounters a state slightly outside the training distribution, it often fails because it has only learned a surface-level mapping rather than the underlying logic of the task.
In many real-world and simulated settings, expert demonstrations are generated by algorithms like A search, Monte Carlo Tree Search (MCTS), or scripted controllers. These algorithms do not just produce an action; they follow a systematic procedure involving intermediate steps, backtracking, and sub-goal evaluation. Procedure Cloning (PC) posits that these intermediate steps are not just "noise" to be discarded, but essential data that explains why* an action was chosen. By supervising the agent on these internal traces, we can train models that "think" before they act.
Methodology: From Actions to Procedures
Standard imitation learning seeks to model the conditional distribution $p(a | s)$. In contrast, Procedure Cloning models the joint distribution of the action and the sequence of intermediate "procedure observations" $\mathbf{x} = (x_1, x_2, ..., x_n)$ that the expert generated while deciding on that action.
Procedure Observations
Procedure observations are snapshots of the expert's internal state. For a search-based expert, these might be the coordinates of nodes currently being expanded in a search tree. For a robotic controller, they might be the target coordinates of a series of waypoints. Crucially, while the expert may use privileged information or tools (like a simulator) to generate these steps, the PC agent only needs to learn to predict the sequence of steps from the initial observation.
Sequence Modeling and Inference
PC treats imitation as a sequence prediction problem, making it highly compatible with modern Transformer architectures. During training, the model is supervised to predict the entire trace of the expert's computation. At inference time, the agent does not have access to the expert's internal tools. Instead, it uses its learned model to autoregressively generate its own internal chain of thought. It "imagines" the search process or the sub-goals internally before committing to a final physical action.
Experimental Results
We evaluated Procedure Cloning across four diverse domains to test its ability to generalize and handle complex tasks.
Navigation and Maze Solving
In a synthetic 32x32 maze environment where the expert uses Breadth-First Search (BFS), standard BC and its variants (including those with auxiliary tasks) achieved 0% success on new, unseen layouts. In contrast, Procedure Cloning achieved 100% success. By predicting the BFS expansion steps, the PC agent effectively learned the logic of pathfinding rather than just memorizing specific routes.
Robotic Manipulation
In the Bimanual Sweep task, two robotic arms must coordinate to sweep particles into bowls. This task requires high precision and handling of privileged information. PC achieved a success metric of 83.9%, significantly outperforming the previous state-of-the-art, Implicit BC, which reached 78.2%. This suggests that imitating the expert's sub-goal generation provides a more robust policy for complex physical coordination.
Strategic Game Playing
Using the MinAtar environment with an MCTS expert, we found that PC agents were remarkably robust. When game difficulty was increased or stochasticity ("sticky actions") was introduced at test time, PC agents maintained high performance. In some instances, the PC agent even outperformed the MCTS expert it was trained on, likely because the sequence modeling process smoothed out the expert's search noise into a more robust representation of game logic.
Discussion and Future Work
Procedure Cloning represents a shift in how we view imitation learning. By treating the expert's internal computations as a "chain of thought," we can leverage the density of supervision provided by modern sequence modeling.
However, PC does introduce new challenges. Generating a full sequence of internal steps for every action is computationally more expensive than a single policy forward pass, which may limit its use in high-frequency real-time control. Additionally, the success of PC depends on the practitioner's ability to identify and record meaningful intermediate steps from the expert. Future work will explore how to automatically discover these procedural traces or use natural language explanations as the "procedure" to further bridge the gap between human reasoning and robotic execution.
Summary updated