Abstract
In this paper, we investigate Exploratory Conservative Policy Optimization (ECPO), a policy optimization strategy that improves exploration behavior while assuring monotonic progress in a principled objective. ECPO conducts maximum entropy exploration within a mirror descent framework, but updates policies using reversed KL projection. This formulation bypasses undesirable mode seeking behavior and avoids premature convergence to sub-optimal policies, while still supporting strong theoretical properties such as guaranteed policy improvement. Experimental evaluations demonstrate that the proposed method significantly improves practical exploration and surpasses the empirical performance of state-of-the art policy optimization methods in a set of benchmark tasks.
Introduction
A central challenge in Deep Reinforcement Learning (RL) is the exploration-exploitation trade-off. Agents often converge prematurely to sub-optimal policies because they stop exploring their environment once a reasonably good strategy is found. This "vicious cycle" occurs when an agent's lack of exploration leads to a lack of reward signal from unexplored regions, further reinforcing the current strategy. While Policy Mirror Descent (PMD) provides a theoretically sound framework for stable updates, it often suffers from "mode-seeking" behavior due to the direction of the Kullback-Leibler (KL) divergence used, which can prematurely collapse the policy distribution.
Exploratory Conservative Policy Optimization (ECPO)
To address these limitations, we propose Exploratory Conservative Policy Optimization (ECPO). ECPO integrates entropy regularization directly into the mirror descent framework to maintain exploration throughout the learning process. The core of the method is a "lift-and-project" procedure.
The Lift-and-Project Framework
The update process is divided into two distinct steps:
- The Lift Step: The current policy is "lifted" to an intermediate ideal policy on the probability simplex. This ideal policy maximizes a combination of the expected reward and an entropy regularizer, ensuring the agent remains "curious" and explores diverse actions.
- The Project Step: This ideal policy is then projected back into the constrained space of policies representable by the agent's neural network.
Reversed KL Divergence and Theoretical Guarantees
A critical innovation in ECPO is the use of reversed KL divergence ($D_{KL}(\pi^* || \pi_\theta)$) during the projection step. Standard methods typically use a "mode-seeking" KL divergence ($D_{KL}(\pi_\theta || \pi^*)$), which tends to fit the neural network to the single most likely action of the target policy. In contrast, the reversed KL is "mean-seeking," encouraging the neural network to cover the entire support of the ideal policy. This prevents the agent from becoming deterministic too early and preserves the diversity of strategies.
Furthermore, the authors prove that for certain common architectures, such as one-layer softmax networks, the projection step can be solved to global optimality. This bridges the gap between simplified theoretical models and practical neural network implementations, providing a mathematical guarantee of policy improvement.
Experimental Evaluation
We evaluated ECPO across a variety of challenging environments, including a large-scale synthetic bandit problem, discrete algorithmic tasks from OpenAI Gym, and continuous control tasks in MuJoCo.
Key Results
- Synthetic Bandit: In a problem with 10,000 actions, ECPO demonstrated superior exploration capabilities. As shown in the Bandit plot, ECPO identifies the optimal action and maintains high performance, whereas baselines like MENT show performance degradation over time.
- Algorithmic Tasks: For tasks requiring precise logical sequences (e.g.,
Reverse,ReversedAddition,RepeatCopy), ECPO showed remarkable stability. While baselines often failed to find a solution or were highly inconsistent in these complex tasks, ECPO consistently achieved high performance. In simpler tasks likeCopyorDuplicatedInput, ECPO reached the same optimal performance as baselines, albeit with a slightly different convergence profile. - Continuous Control: We extended the framework to an actor-critic version, ECAC (Exploratory Conservative Actor-Critic). In MuJoCo benchmarks, ECAC matched or outperformed state-of-the-art algorithms like Soft Actor-Critic (SAC) and TD3, particularly in complex environments like
AntandHumanoidwhere it demonstrated better sample efficiency.
Conclusion
ECPO provides a principled approach to entropy-regularized policy optimization. By leveraging the "lift-and-project" framework and reversed KL divergence, it ensures both monotonic policy improvement and robust exploration. While achieving global optimality in the projection step remains a challenge for very deep architectures, ECPO demonstrates significant empirical gains over existing methods in both discrete and continuous domains.
Summary updated