TL;DR
This paper introduces Symbolic Imitation Learning (SIL), a framework that distills complex "black-box" reinforcement learning policies into simple, human-readable decision lists. By combining offline imitation learning with online stochastic tuning (Augmented Random Search), the authors produced policies that not only match or exceed the performance of their "oracle" teachers in 11 out of 20 benchmark tasks but also remain small enough for a domain expert to actually understand.
What Problem Does This Paper Address?
Deep Reinforcement Learning (DRL) has become the gold standard for continuous control, but it has a major transparency problem: the resulting policies are massive neural networks that offer zero insight into why an agent makes a specific decision. Conversely, interpretable methods like Genetic Programming (GP) often struggle to reach the same performance levels in complex, high-dimensional environments.
The authors aim to bridge this gap. They tackle the challenge of creating policies that are parsimonious (simple), interpretable, and high-performing. They specifically address the issue of "distributional drift" in offline learning—where an agent trained on static data fails when it encounters slightly different states in the real world—by introducing an online tuning phase.
Methods
The SIL framework operates in two distinct stages:
- Offline Phase (Structure Discovery):
- Motion Discovery: The system takes demonstrations from an "oracle" (a pre-trained DRL agent or a human) and uses an Expectation-Maximization (EM) algorithm to cluster actions into a few "motion primitives"—simple linear models that describe different modes of behavior (e.g., "swinging a leg" vs. "balancing").
- Switching Penalty: To prevent the agent from jittering between behaviors, they use a dynamic programming step that penalizes switching motion primitives too frequently.
- Decision List Evolution: Genetic Programming is used to evolve an
if-elif-elsestructure (a decision list) that decides which motion primitive to use based on the current state. - Online Phase (Fine-Tuning):
- ARS Tuning: Once the structure is fixed, the framework uses Augmented Random Search (ARS)—a simple but effective derivative-free optimizer—to fine-tune the constants within the decision rules and the weights of the linear motion primitives through direct interaction with the environment.
Datasets
The authors evaluated their framework across 20 benchmark RL tasks, providing a comprehensive look at how it scales from simple to complex environments:
- MuJoCo: Classic continuous control tasks like Hopper, Walker2d, Ant, and the high-dimensional Humanoid.
- Classic Control: CartPole, MountainCar, and Acrobot.
- Box2D: LunarLander and BipedalWalker.
- Robotics Benchmarks: High-dimensional tasks including the Adroit Hand (manipulating pens, hammers, and doors) and Franka Kitchen (manipulating household items).
The "oracles" providing the training data included a mix of DRL algorithms (TD3, SAC, PPO, TQC) and human demonstrations.
Key Insights and Ideas
- Complexity is often Overkill: The paper demonstrates that even for a task as complex as the 376-dimensional Humanoid, the underlying logic can be represented by a relatively small set of rules and linear motions. DRL networks are often far more complex than the task actually requires.
- Structure vs. Parameters: A key "aha moment" is the realization that offline learning is excellent for finding the logic (the decision list), but online interaction is crucial for perfecting the execution (the constants). Offline learning alone rarely "solved" the harder tasks.
- Smoothness via Penalties: By introducing a switching penalty during motion discovery, the authors created policies that are not only more interpretable but also more physically realistic, avoiding high-frequency "chatter" between different control modes.
Key Results
- Superior Performance: The SIL framework exceeded the performance of the original DRL oracle in 11 out of 20 tasks.
- State-of-the-Art for EC: Compared to other evolutionary computation (EC) methods like MAPLE or CGP, SIL consistently provided higher returns, particularly on the more difficult MuJoCo and robotics tasks.
- Extreme Parsimony: The evolved policies are remarkably small. For example, the Adroit Hammer task was solved using a policy with only two predicates and three motion primitives, yet it achieved a score of over 18,000.
Limitations and Future Work
While SIL is powerful, it currently relies on having a high-quality oracle to provide initial demonstrations. If the oracle is poor, the discovered motion primitives may be suboptimal.
The authors also note that they are not currently utilizing the reward signal during the offline phase, only the state-action pairs. Future work could involve using these rewards to mitigate distribution shift more effectively or exploring "sub-trajectory stitching" to combine the best parts of multiple imperfect demonstrations. Finally, exploring whether more efficient parameter tuning methods (like xNES) could replace ARS remains an open question.
Summary updated