Fast Policy Learning through Imitation and Reinforcement

Ching-An Cheng, Xinyan Yan, Nolan Wagener, Byron Boots

Abstract

LOKI (Locally Optimal search after K-step Imitation) is a simple yet powerful "imitate-then-reinforce" strategy that bridges the gap between fast-but-limited imitation learning and slow-but-thorough reinforcement learning. By unifying both fields under a single Mirror Descent framework and using a randomized switching time, LOKI consistently outperforms suboptimal experts and converges significantly faster than standard reinforcement learning.

Introduction

Reinforcement Learning (RL) is capable of discovering superhuman policies but is notoriously sample-hungry, often requiring millions of interactions for even simple tasks. Conversely, Imitation Learning (IL) is much faster because it leverages expert demonstrations, but it is typically limited by the expert's performance ceiling. If an expert is suboptimal, standard IL will plateau.

Previous attempts to combine RL and IL often required "state resetting"—the ability to teleport an agent to a specific state to query the expert—which is often impossible in real-world robotics. LOKI addresses these challenges by providing a theoretically sound method to surpass mediocre experts without requiring state resets or complex value-function estimations.

Methodology: A Unified Framework

The core contribution of this work is the theoretical unification of RL and IL. The authors demonstrate that popular algorithms, such as Policy Gradient and DAgger, are variations of the same mathematical framework: Mirror Descent. The primary distinction lies in the "first-order oracle" (the gradient estimate) employed by each method. RL typically uses a gradient of the expected return, while IL uses a gradient of a surrogate loss.

First-order oracles for various RL and IL methods under the Mirror Descent framework.

The LOKI Algorithm

Building on this insight, LOKI operates in two distinct phases:

  1. Imitation Phase: For a randomly chosen number of steps $K$, the agent performs online imitation learning, minimizing the KL-divergence between its policy and the expert's.
  2. Reinforcement Phase: The agent then switches to a standard RL policy gradient method (like TRPO) to refine the policy and surpass the expert.

The switching time $K$ is sampled from a specific probability distribution. This randomization allows the authors to prove that LOKI performs as if it were running RL directly from the expert's starting point, providing strong convergence guarantees.

Experimental Evaluation

LOKI was evaluated on several robotic control benchmarks from OpenAI Gym using the DART physics engine, including Inverted Pendulum, Reacher, and locomotion tasks such as Hopper, 2D Walker, and 3D Walker.

Key Results

  • Efficiency and Performance: In the 2D Walker task, LOKI reached the suboptimal expert's performance in just ~15 iterations and eventually doubled the expert's reward score. Pure RL (TRPO) took significantly longer to reach comparable levels.
  • Robustness in High Dimensions: LOKI significantly outperformed more complex hybrid algorithms like THOR and SLOLS in tasks like the 3D Walker. This is attributed to LOKI's avoidance of expert value-function estimation, which often fails in high-dimensional spaces due to covariate shift.
  • Seamless Transition: Experimental curves show that LOKI transitions smoothly from the imitation phase to the reinforcement phase without the performance "dip" often seen when switching algorithms.

Discussion and Future Work

LOKI treats the expert as a "warm-start" rather than an absolute truth, allowing the agent to quickly reach the expert's neighborhood and then use RL to fine-tune its way to a superior solution.

However, the method currently requires a "queryable" expert policy and exhibits sensitivity to the range chosen for the randomized switching time $K$. Future directions include adapting LOKI to work with purely offline data and deploying it on complex, real-world robotic systems where sample efficiency is the primary barrier to success.

Summary updated

Abstract

Imitation learning (IL) consists of a set of tools that leverage expert demonstrations to quickly learn policies. However, if the expert is suboptimal, IL can yield policies with inferior performance compared to reinforcement learning (RL). In this paper, we aim to provide an algorithm that combines the best aspects of RL and IL. We accomplish this by formulating several popular RL and IL algorithms in a common mirror descent framework, showing that these algorithms can be viewed as a variation on a single approach. We then propose LOKI, a strategy for policy learning that first performs a small but random number of IL iterations before switching to a policy gradient RL method. We show that if the switching time is properly randomized, LOKI can learn to outperform a suboptimal expert and converge faster than running policy gradient from scratch. Finally, we evaluate the performance of LOKI experimentally in several simulated environments.
BibTeX
@inproceedings{cheng2018fast,
  author = {Ching-An Cheng and Xinyan Yan and Nolan Wagener and Byron Boots},
  title = {{Fast Policy Learning through Imitation and Reinforcement}},
  booktitle = {Conference on Uncertainty in Artificial Intelligence},
  year = {2018},
  url = {https://openreview.net/forum?id=lqkSNRIlOJ},
}
Figure