Making Strides in Symbolic Imitation Learning: Offline Demonstrations, Online Tuning

Bryce MacInnis, Malcolm I. HeywoodarXiv 2026 · arXiv

TL;DR

This paper introduces Symbolic Imitation Learning (SIL), a framework that distills complex "black-box" reinforcement learning policies into simple, human-readable decision lists. By combining offline imitation learning with online stochastic tuning (Augmented Random Search), the authors produced policies that not only match or exceed the performance of their "oracle" teachers in 11 out of 20 benchmark tasks but also remain small enough for a domain expert to actually understand.

What Problem Does This Paper Address?

Deep Reinforcement Learning (DRL) has become the gold standard for continuous control, but it has a major transparency problem: the resulting policies are massive neural networks that offer zero insight into why an agent makes a specific decision. Conversely, interpretable methods like Genetic Programming (GP) often struggle to reach the same performance levels in complex, high-dimensional environments.

The authors aim to bridge this gap. They tackle the challenge of creating policies that are parsimonious (simple), interpretable, and high-performing. They specifically address the issue of "distributional drift" in offline learning—where an agent trained on static data fails when it encounters slightly different states in the real world—by introducing an online tuning phase.

Methods

The SIL framework operates in two distinct stages:

  1. Offline Phase (Structure Discovery):
    • Motion Discovery: The system takes demonstrations from an "oracle" (a pre-trained DRL agent or a human) and uses an Expectation-Maximization (EM) algorithm to cluster actions into a few "motion primitives"—simple linear models that describe different modes of behavior (e.g., "swinging a leg" vs. "balancing").
  2. Switching Penalty: To prevent the agent from jittering between behaviors, they use a dynamic programming step that penalizes switching motion primitives too frequently.
  3. Decision List Evolution: Genetic Programming is used to evolve an if-elif-else structure (a decision list) that decides which motion primitive to use based on the current state.
  4. Online Phase (Fine-Tuning):
    • ARS Tuning: Once the structure is fixed, the framework uses Augmented Random Search (ARS)—a simple but effective derivative-free optimizer—to fine-tune the constants within the decision rules and the weights of the linear motion primitives through direct interaction with the environment.
Fig. 1. Overall framework for online/offline evolution of a decision list with (linear) motion primitives. Only the ARS tuning step is an online process

Datasets

The authors evaluated their framework across 20 benchmark RL tasks, providing a comprehensive look at how it scales from simple to complex environments:

  • MuJoCo: Classic continuous control tasks like Hopper, Walker2d, Ant, and the high-dimensional Humanoid.
  • Classic Control: CartPole, MountainCar, and Acrobot.
  • Box2D: LunarLander and BipedalWalker.
  • Robotics Benchmarks: High-dimensional tasks including the Adroit Hand (manipulating pens, hammers, and doors) and Franka Kitchen (manipulating household items).

The "oracles" providing the training data included a mix of DRL algorithms (TD3, SAC, PPO, TQC) and human demonstrations.

Table 2. Summary of oracle datasets. Oracle denotes the source of the offline dataset. Rows is the cardinality for the offline dataset. |State| and |Action| are the dimension of the state (input) and action (output) respectively

Key Insights and Ideas

  • Complexity is often Overkill: The paper demonstrates that even for a task as complex as the 376-dimensional Humanoid, the underlying logic can be represented by a relatively small set of rules and linear motions. DRL networks are often far more complex than the task actually requires.
  • Structure vs. Parameters: A key "aha moment" is the realization that offline learning is excellent for finding the logic (the decision list), but online interaction is crucial for perfecting the execution (the constants). Offline learning alone rarely "solved" the harder tasks.
  • Smoothness via Penalties: By introducing a switching penalty during motion discovery, the authors created policies that are not only more interpretable but also more physically realistic, avoiding high-frequency "chatter" between different control modes.
Fig. 2. An illustrated example of the optional dynamic-programming step when the switching penalty is used to reduce the amount of switching between motion primitives despite incurring slightly greater cost without the penalty.

Key Results

  • Superior Performance: The SIL framework exceeded the performance of the original DRL oracle in 11 out of 20 tasks.
  • State-of-the-Art for EC: Compared to other evolutionary computation (EC) methods like MAPLE or CGP, SIL consistently provided higher returns, particularly on the more difficult MuJoCo and robotics tasks.
  • Extreme Parsimony: The evolved policies are remarkably small. For example, the Adroit Hammer task was solved using a policy with only two predicates and three motion primitives, yet it achieved a score of over 18,000.
Table 1. Summary of comparator method results on benchmark RL tasks. Values for MAPLE, CGP and xNES+ had to be estimated visually from figures. CMA-ES, SIRL and ARS are the average test performance quoted from the original works. †indicates a result from ARS v1 versus v2. CMA-ES results are quoted as the best from 6 different versions [20]

Limitations and Future Work

While SIL is powerful, it currently relies on having a high-quality oracle to provide initial demonstrations. If the oracle is poor, the discovered motion primitives may be suboptimal.

The authors also note that they are not currently utilizing the reward signal during the offline phase, only the state-action pairs. Future work could involve using these rewards to mitigate distribution shift more effectively or exploring "sub-trajectory stitching" to combine the best parts of multiple imperfect demonstrations. Finally, exploring whether more efficient parameter tuning methods (like xNES) could replace ARS remains an open question.

Summary updated

Abstract

Deep reinforcement learning (DRL) methods have been the dominant paradigm in continuous control tasks due to their efficacy in function approximation, while interpretable evolutionary methods have lagged largely behind for the opposite reason. In this work, we introduce Symbolic Imitation Learning, a framework for distilling an oracle policy through an offline dataset into a decision list with linear model motion primitives. We show that this, in conjunction, with online Augmented-Random Search (ARS) tuning of the constants and motion primitives approximates the performance of the oracle across 20 benchmark RL tasks and exceeds the oracle in 11 of those 20 tasks while simultaneously producing parsimonious, interpretable policies that domain experts can better understand.
BibTeX
@misc{macinnis2026making,
  author = {Bryce MacInnis and Malcolm I. Heywood},
  title = {{Making Strides in Symbolic Imitation Learning: Offline Demonstrations, Online Tuning}},
  year = {2026},
  eprint = {2611.13182},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.13182},
}
Figure