Abstract
We consider the problem of learning the optimal policy of an unknown Markov decision process (MDP) when expert demonstrations are available along with interaction samples. We build on classification-based policy iteration to perform a seamless integration of interaction and expert data, thus obtaining an algorithm which can benefit from both sources of information at the same time. Furthermore, we provide a full theoretical analysis of the performance across iterations providing insights on how the algorithm works. Finally, we report an empirical evaluation of the algorithm and a comparison with the state-of-the-art algorithms.
Introduction
In Reinforcement Learning (RL), agents typically face a trade-off between learning from scratch through trial and error (pure RL) and mimicking an expert (Imitation Learning). While pure RL is often computationally expensive and slow to converge due to exploration challenges, Imitation Learning is limited by the quality and quantity of expert demonstrations.
Standard Approximate Policy Iteration (API) algorithms attempt to improve policies by evaluating current actions and finding a "greedy" improvement. However, combining these with expert data has historically been difficult, often resulting in complex constraints or non-convex optimization problems that lack theoretical guarantees. This paper introduces Direct Policy Iteration with Demonstrations (DPID), a framework that treats policy improvement as a joint cost-sensitive classification task, seamlessly blending interaction and expert data.
Methodology
DPID builds upon Direct Policy Iteration (DPI), which uses rollouts to identify superior actions and trains a classifier to predict the optimal policy. The core innovation of DPID is the modification of the classifier's training set to include both RL samples (weighted by the "cost of error") and expert demonstration samples.
A hyperparameter, $\alpha$, is introduced to control the "trust" level in the expert data. By minimizing a joint loss function ($L_{total} = L_{RL} + \alpha L_{Expert}$), the agent balances greedy RL improvement with expert imitation. This allows the agent to use expert data to "jumpstart" learning while retaining the ability to eventually surpass the expert through environment interaction.
Experimental Results
The researchers evaluated DPID on the Garnet Domain (synthetic MDPs) and a Vehicle Brake Control task.
As shown in the experimental results, DPID consistently outperforms both pure RL (DPI, where $\alpha=0$) and pure Imitation Learning (SVM). Key findings include:
- Performance and Speed: DPID reaches near-optimal performance significantly faster than pure RL and surpasses the performance of the expert demonstrations used in SVM.
- Robustness to Alpha: The algorithm is relatively robust to the choice of $\alpha$. Even a small nudge from expert data (e.g., $\alpha/N_E = 0.01$) provides significant benefits over pure RL.
- Suboptimal Experts: Even when experts are mediocre, DPID can utilize their data to narrow the search space in early iterations without becoming trapped at the expert's performance level.
Discussion and Insights
One of the most significant theoretical findings is that the performance loss of DPID is bounded by the minimum of the expert's error and the RL propagation error. This ensures the algorithm effectively takes the best available information from both sources. Furthermore, DPID proved robust to feature bias; even with poor state representations, expert data acted as a stabilizer to keep learning on track.
Unlike previous methods that require solving constrained optimization problems, DPID is computationally simple, requiring only a standard cost-sensitive classifier. It performs comparably to or better than state-of-the-art algorithms like DAgger, without requiring the expert to be available for real-time labeling during training.
Conclusion and Future Work
DPID offers a robust, theoretically grounded method for integrating demonstrations into RL. Future research may explore adaptive $\alpha$ values—starting high to leverage experts and decaying as the agent gains experience—and extending the full theoretical analysis to larger, discrete action spaces beyond the binary cases focused on in this study.
Summary updated