Better Exploration with Optimistic Actor-Critic

Kamil Ciosek, Quan Vuong, Robert Loftin, Katja Hofmann

Abstract

Actor-critic methods, a type of model-free Reinforcement Learning, have been successfully applied to challenging tasks in continuous control, often achieving state-of-the art performance. However, wide-scale adoption of these methods in real-world domains is made difficult by their poor sample efficiency. We address this problem both theoretically and empirically. On the theoretical side, we identify two phenomena preventing efficient exploration in existing state-of-the-art algorithms such as Soft Actor Critic. First, combining a greedy actor update with a pessimistic estimate of the critic leads to the avoidance of actions that the agent does not know about, a phenomenon we call pessimistic underexploration. Second, current algorithms are directionally uninformed, sampling actions with equal probability in opposite directions from the current mean. This is wasteful, since we typically need actions taken along certain directions much more than others. To address both of these phenomena, we introduce a new algorithm, Optimistic Actor Critic, which approximates a lower and upper confidence bound on the state-action value function. This allows us to apply the principle of optimism in the face of uncertainty to perform directed exploration using the upper bound while still using the lower bound to avoid overestimation. We evaluate OAC in several challenging continuous control tasks, achieving state-of the art sample efficiency.

Introduction

Reinforcement Learning (RL) has demonstrated remarkable success in complex domains, yet its practical application—particularly in robotics—is often limited by extreme "sample hunger." State-of-the-art algorithms like Soft Actor-Critic (SAC) and Twin Delayed Deep Deterministic Policy Gradient (TD3) often require millions of environment interactions to master even relatively simple continuous control tasks. This paper investigates the theoretical underpinnings of this inefficiency and proposes a novel solution: Optimistic Actor-Critic (OAC).

Identifying the Barriers to Exploration

The authors identify two primary flaws in current actor-critic methods that significantly impede exploration efficiency:

  1. Pessimistic Underexploration: To maintain stability and prevent overestimation bias, modern algorithms utilize a "lower bound" estimate of the state-action value function. However, if this estimate is erroneously low in a promising but poorly explored region, the agent may prematurely conclude the area is low-value and cease exploration there entirely.
  2. Directional Uninformedness: Standard exploration strategies typically sample actions symmetrically around the current policy mean. This is inherently wasteful; if an agent has already learned that a specific direction is suboptimal, it should prioritize exploration in the opposite, potentially more rewarding direction rather than sampling both equally.
Figure 1: Illustration of (a) pessimistic underexploration and (b) directional uninformedness.

Methodology: Optimistic Actor-Critic (OAC)

OAC addresses these issues by decoupling the policy used for exploration from the policy used for learning. The core innovation lies in leveraging the principle of "optimism in the face of uncertainty."

Epistemic Uncertainty and Bounds

OAC maintains two critic networks to estimate the state-action value function. The variance between these networks serves as a proxy for "epistemic uncertainty"—uncertainty arising from a lack of data. Using this, the algorithm computes both a Lower Confidence Bound (LCB) for stable learning and an Upper Confidence Bound (UCB) for directed exploration.

The Optimistic Explorer

For data collection, OAC constructs a temporary exploration policy ($\pi_E$). This policy shifts its mean toward the gradient of the UCB, effectively guiding the agent toward regions of high potential value or high uncertainty. To ensure safety and stability, this shift is constrained by a Kullback–Leibler (KL) divergence limit, keeping the exploration policy within a reasonable distance of the current target policy.

The Pessimistic Learner

While exploration is optimistic, the target policy update remains pessimistic. By using the LCB for the actor update, OAC retains the stability benefits of SAC and TD3, ensuring that the learned policy does not over-commit to over-optimistic value estimates.

Computational Efficiency

Notably, OAC achieves these performance gains with negligible computational overhead. Since modern algorithms like SAC already employ dual critic networks, calculating the uncertainty and the resulting shifted mean for the exploration policy is essentially "free" in terms of processing time.

Experimental Evaluation

The performance of OAC was benchmarked against the MuJoCo (Multi-Joint dynamics with Contact) suite, including tasks such as Ant-v2, Hopper-v2, Walker2d-v2, HalfCheetah-v2, and the highly complex Humanoid-v2.

Results

OAC achieved state-of-the-art sample efficiency across the benchmark suite. Notably, on the challenging Humanoid-v2 task, OAC reached high performance levels approximately 1.8 times faster than the standard SAC algorithm.

Figure 5: Performance comparison on the Humanoid-v2 task, demonstrating OAC's superior sample efficiency compared to SAC.

Ablation studies confirmed that the bootstrapped uncertainty estimate was the primary driver of these gains, particularly in complex environments. Furthermore, the algorithm proved robust across different random seeds and showed low sensitivity to the KL-divergence hyperparameter ($\delta$).

Conclusion and Future Work

Optimistic Actor-Critic provides a theoretically grounded and computationally efficient method for improving exploration in continuous control. By balancing optimism for discovery with pessimism for stability, OAC significantly reduces the sample requirements for complex RL tasks. Future work may explore more sophisticated uncertainty estimation techniques beyond simple two-network bootstrapping and investigate the algorithm's performance in environments with higher reward variance.

Summary updated

Abstract

Actor-critic methods, a type of model-free Reinforcement Learning, have been successfully applied to challenging tasks in continuous control, often achieving state-of-the art performance. However, wide-scale adoption of these methods in real-world domains is made difficult by their poor sample efficiency. We address this problem both theoretically and empirically. On the theoretical side, we identify two phenomena preventing efficient exploration in existing state-of-the-art algorithms such as Soft Actor Critic. First, combining a greedy actor update with a pessimistic estimate of the critic leads to the avoidance of actions that the agent does not know about, a phenomenon we call pessimistic underexploration. Second, current algorithms are directionally uninformed, sampling actions with equal probability in opposite directions from the current mean. This is wasteful, since we typically need actions taken along certain directions much more than others. To address both of these phenomena, we introduce a new algorithm, Optimistic Actor Critic, which approximates a lower and upper confidence bound on the state-action value function. This allows us to apply the principle of optimism in the face of uncertainty to perform directed exploration using the upper bound while still using the lower bound to avoid overestimation. We evaluate OAC in several challenging continuous control tasks, achieving state-of the art sample efficiency.
BibTeX
@inproceedings{ciosek2019better,
  author = {Kamil Ciosek and Quan Vuong and Robert Loftin and Katja Hofmann},
  title = {{Better Exploration with Optimistic Actor-Critic}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2019},
  url = {https://openreview.net/forum?id=un-AbhNjiG},
}
Figure