Abstract
Imitation learning (IL) consists of a set of tools that leverage expert demonstrations to quickly learn policies. However, if the expert is suboptimal, IL can yield policies with inferior performance compared to reinforcement learning (RL). In this paper, we aim to provide an algorithm that combines the best aspects of RL and IL. We accomplish this by formulating several popular RL and IL algorithms in a common mirror descent framework, showing that these algorithms can be viewed as a variation on a single approach. We then propose LOKI, a strategy for policy learning that first performs a small but random number of IL iterations before switching to a policy gradient RL method. We show that if the switching time is properly randomized, LOKI can learn to outperform a suboptimal expert and converge faster than running policy gradient from scratch. Finally, we evaluate the performance of LOKI experimentally in several simulated environments.
Introduction
Training agents for complex sequential decision-making tasks typically involves a choice between two paradigms: Reinforcement Learning (RL) and Imitation Learning (IL). RL algorithms, such as Policy Gradients, are capable of discovering optimal policies from scratch but are notoriously sample-inefficient, often requiring millions of environment interactions. Conversely, IL leverages expert demonstrations to achieve rapid progress. However, IL is fundamentally limited by the quality of the expert; if the demonstrator is suboptimal, the learner will likely plateau at that same level of performance.
While hybrid methods like AGGREVATE or THOR have attempted to bridge this gap, they often rely on strong assumptions, such as the ability to reset the environment to arbitrary states or the accurate estimation of the expert's value function. In high-dimensional spaces, these estimations are prone to error due to covariate shift. This paper introduces LOKI (Locally Optimal search after K-step Imitation), a mathematically grounded approach that uses a suboptimal expert to "warm-start" learning before transitioning to RL to achieve optimality.
A Unified Framework for RL and IL
A key contribution of this work is the theoretical unification of RL and IL. The authors demonstrate that first-order RL (e.g., Policy Gradients) and online IL (e.g., DAgger) can both be viewed as instances of mirror descent. In this framework, the primary difference between algorithms lies in the choice of the "first-order oracle"—the specific gradient update direction used to improve the policy.
As shown in the table above, while the underlying optimization structure is shared, the oracles differ. RL oracles focus on monotonic improvement based on the learner's current advantage, whereas IL oracles focus on minimizing the divergence from the expert's policy. This insight allows for a seamless transition between the two modes of learning.
The LOKI Algorithm
Building on the mirror descent unification, the authors propose LOKI. The algorithm is designed to be simple and robust, operating in two distinct phases:
- Imitation Phase: The algorithm first samples a random integer $K$ from a specific probability distribution. It then executes $K$ steps of online imitation learning. During this phase, the loss function is typically the KL-divergence between the expert's policy and the learner's policy.
- Reinforcement Phase: After $K$ steps, the algorithm switches to a standard policy gradient RL method (such as TRPO with Generalized Advantage Estimation).
The randomization of the switching point $K$ is a critical theoretical component. The authors prove that this randomization bounds the performance gap, ensuring that LOKI performs as well as if it had been initialized perfectly at the expert's policy before starting RL. Furthermore, during the imitation phase, LOKI collects environment data to "warm up" the on-policy value function, ensuring a smooth transition to the advantage-based updates of the reinforcement phase.
Experimental Evaluation
LOKI was evaluated on a variety of simulated robotic control tasks from OpenAI Gym using the DART physics engine, including Inverted Pendulum, Hopper, 2D/3D Walker, and a 5-DOF Reacher arm. To test the algorithm's ability to surpass suboptimal demonstrators, experts were generated by intentionally stopping TRPO training before convergence.
Key Results
- Performance and Efficiency: Across all tasks, LOKI's learning curve matched an "Ideal" baseline (RL initialized at the expert's policy). It significantly outperformed pure RL (which was slow) and pure IL (which plateaued at the expert's suboptimal level).
- Robustness to Complexity: In simple tasks, prior hybrid methods like SLOLS and THOR performed adequately. However, in complex locomotion tasks (e.g., 3D Walker), these methods often collapsed. This failure is attributed to their reliance on estimating the expert's value function, which suffers from covariate shift as the agent deviates from the expert's demonstrations. LOKI avoids this by using KL-divergence for imitation and on-policy advantage for RL, maintaining dominance in high-dimensional environments.
Conclusion and Future Work
LOKI provides a simple yet effective strategy for leveraging suboptimal experts to accelerate reinforcement learning. By framing the problem within a unified mirror descent framework and employing a randomized switching strategy, it achieves the speed of imitation learning without being capped by the expert's performance.
A current limitation is LOKI's reliance on online imitation learning, which requires an interactive expert oracle. Future research could explore extending this randomized switching framework to offline settings, where only a static dataset of demonstrations is available.
Summary updated