Dual Policy Iteration

Wen Sun, Geoffrey J. Gordon, Byron Boots, J. Andrew Bagnell

Abstract

Recently, a novel class of Approximate Policy Iteration (API) algorithms have demonstrated impressive practical performance (e.g., ExIt, AlphaGo-Zero). This new family of algorithms maintains, and alternately optimizes, two policies: a fast, reactive policy (e.g., a deep neural network) deployed at test time, and a slow, non-reactive policy (e.g., Tree Search), that can plan multiple steps ahead. The reactive policy is updated under supervision from the non-reactive policy, while the non-reactive policy is improved with guidance from the reactive policy. In this work we study this Dual Policy Iteration (DPI) strategy in an alternating optimization framework and provide a convergence analysis that extends existing API theory. We also develop a special instance of this framework which reduces the update of non-reactive policies to model-based optimal control using learned local models, and provides a theoretically sound way of unifying model-free and model-based RL approaches with unknown dynamics. We demonstrate the efficacy of our approach on various continuous control Markov Decision Processes.

Introduction

Reinforcement Learning (RL) has traditionally been divided into two primary paradigms: model-free and model-based. Model-free methods, such as Policy Gradient, often suffer from extreme sample inefficiency due to their reliance on random exploration. Conversely, model-based methods attempt to learn a global model of the environment, which can be biased or prohibitively data-intensive to train across the entire state space.

Recent breakthroughs like AlphaGo-Zero and Expert Iteration (ExIt) have suggested a powerful middle ground: using a slow, deliberate planner to "teach" a fast, reactive neural network. This paper formalizes this "teacher-student" approach as Dual Policy Iteration (DPI), providing the theoretical foundation and a practical, sample-efficient algorithm for continuous control even when environment dynamics are unknown.

The Dual Policy Iteration Framework

DPI operates through an alternating optimization strategy involving two distinct policies:

  1. Reactive Policy ($\pi$): A fast function approximator (e.g., a deep neural network) designed for efficient execution at test time.
  2. Intermediate Expert ($\eta$): A slower, more capable policy (e.g., Tree Search or Model-Based Optimal Control) used exclusively during the training phase to guide the reactive policy.

The DPI loop consists of three main stages:

  • Local Model Fitting: The agent executes the current reactive policy $\pi$ to collect interaction data and fits a local dynamics model. A key insight of DPI is that the model only needs to be accurate in the regions of the state space currently visited by the policy.
  • Expert Planning: Using the learned local model, the agent performs systematic planning (e.g., via Model-Based Optimal Control) to derive an improved local expert policy $\eta$.
  • Policy Update (Imitation): The reactive policy $\pi$ is updated to imitate the expert $\eta$ using an imitation learning objective, such as the AGGREVATED update rule.

By constraining the expert to stay within a "trust region" of the reactive policy, DPI ensures stable and monotonic performance improvements throughout the learning process.

Experimental Evaluation

The authors evaluated DPI across a diverse set of benchmarks, ranging from discrete MDPs to complex continuous control tasks in the MuJoCo simulator and high-fidelity helicopter aerobatics (Hover and Funnel tasks).

Performance comparison of DPI (AggreVaTeD) vs. baselines across discrete and continuous control tasks.

As shown in Figure 1, the DPI-based implementation (AggreVaTeD_iLQR) consistently outperforms strong model-free baselines like TRPO-GAE. In tasks such as Cart-Pole, Helicopter Hover, and various MuJoCo environments (Swimmer, Hopper, Half-Cheetah), DPI achieves significantly higher sample efficiency, reaching optimal performance levels (lower cost/return) with far fewer environmental interactions.

Key Insights and Discussion

  • Efficiency through Imitation: Updating a policy by imitating a superior planner is mathematically more efficient than learning from raw rewards. DPI leverages the expert's advantage function ($A^\eta$) to provide a stronger learning signal than the standard policy advantage ($A^\pi$).
  • Local vs. Global Modeling: DPI demonstrates that a perfect global model is unnecessary for policy improvement. Local accuracy under the current state-action distribution is sufficient to derive a better expert.
  • Robustness: Beyond standard benchmarks, DPI shows superior performance in "robust" settings where physical parameters vary across environments, avoiding the overfitting often seen in non-robust model-free approaches.

Conclusion

Dual Policy Iteration provides a unifying theoretical framework for a new class of highly successful RL algorithms. By combining the strengths of local model-based planning with the representational power of deep reactive policies, DPI offers a path toward more sample-efficient and robust autonomous systems. While computationally more intensive per iteration than some model-free methods, its dramatic reduction in required environment steps makes it a compelling choice for real-world applications where data collection is expensive.

Summary updated

Abstract

Recently, a novel class of Approximate Policy Iteration (API) algorithms have demonstrated impressive practical performance (e.g., ExIt from [2], AlphaGo-Zero from [27]). This new family of algorithms maintains, and alternately optimizes, two policies: a fast, reactive policy (e.g., a deep neural network) deployed at test time, and a slow, non-reactive policy (e.g., Tree Search), that can plan multiple steps ahead. The reactive policy is updated under supervision from the non-reactive policy, while the non-reactive policy is improved with guidance from the reactive policy. In this work we study this Dual Policy Iteration (DPI) strategy in an alternating optimization framework and provide a convergence analysis that extends existing API theory. We also develop a special instance of this framework which reduces the update of non-reactive policies to model-based optimal control using learned local models, and provides a theoretically sound way of unifying model-free and model-based RL approaches with unknown dynamics. We demonstrate the efficacy of our approach on various continuous control Markov Decision Processes.
BibTeX
@inproceedings{sun2018dual,
  author = {Wen Sun and Geoffrey J. Gordon and Byron Boots and J. Andrew Bagnell},
  title = {{Dual Policy Iteration}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2018},
  url = {https://openreview.net/forum?id=ryWoLOZd-H},
}
Figure