Better Exploration with Optimistic Actor Critic

Kamil Ciosek, Quan Vuong, Robert Loftin, Katja Hofmann

TL;DR

Optimistic Actor-Critic (OAC) significantly improves the sample efficiency of reinforcement learning in continuous control tasks. By using an optimistic upper bound of the value function to guide exploration while maintaining a conservative lower bound for stable updates, OAC overcomes the "pessimistic underexploration" inherent in current state-of-the-art algorithms like Soft Actor-Critic (SAC).

What Problem Does This Paper Address?

Reinforcement Learning (RL) has seen massive success in simulated environments, but its "sample inefficiency"—the need for millions of interactions to learn a simple task—makes it prohibitively expensive for real-world robotics.

The authors identify two specific flaws in current leading algorithms (like SAC and TD3) that cause this inefficiency:

  1. Pessimistic Underexploration: To prevent the agent from overestimating how good an action is (a common failure mode), these algorithms use a "pessimistic" lower-bound estimate of rewards. However, this causes the agent to avoid any action it is uncertain about, effectively killing curiosity.
  2. Directional Uninformedness: Standard algorithms sample actions by adding random noise in all directions equally. This is wasteful because the agent often already knows that certain directions are bad; it should focus its exploration on directions where the potential for high reward is most uncertain.

Methods

OAC introduces a novel way to balance the "deadly triad" of function approximation, bootstrapping, and off-policy learning. Its core components are:

  • Bootstrapped Uncertainty: OAC maintains two separate Q-networks (critics). By looking at the difference between these two networks, the agent can estimate epistemic uncertainty—essentially measuring "how much do I not know about this state-action pair?"
  • The Optimistic Exploration Policy ($\pi_E$): Unlike standard methods that use the same policy for both acting and learning, OAC creates a separate exploration policy. This policy is shifted toward the Upper Confidence Bound (UCB) of the Q-function. It moves the mean of the action distribution toward areas where the reward might be high.
OAC uses an upper bound for exploration and a lower bound for the target policy.
  • KL-Constraint for Stability: To ensure the agent doesn't take "catastrophically bad" actions while exploring, the exploration policy is constrained to stay within a certain "distance" (measured by KL-divergence) from the target policy.
  • Conservative Target Policy: While exploration is optimistic, the actual learning update still uses a lower bound to maintain the stability benefits of algorithms like SAC.

Datasets

The authors evaluated OAC on the MuJoCo (Multi-Joint dynamics with Contact) continuous control benchmarks, which are the industry standard for simulated robotics. Specifically, they tested on:

  • Ant-v2
  • Hopper-v2
  • Walker2d-v2
  • HalfCheetah-v2
  • Humanoid-v2 (The most complex and challenging of the set)

Key Insights and Ideas

  • Optimism in the Face of Uncertainty: This is a classic RL principle usually applied to discrete tasks (like multi-armed bandits). OAC successfully adapts this to the high-dimensional, continuous world of deep actor-critics.
  • Decoupling Exploration from Exploitation: By using one policy to collect data ($\pi_E$) and another to define the "best" behavior ($\pi_T$), OAC can be aggressive in its search for rewards without making the final learned behavior unstable.
  • Directed Exploration: Because the exploration policy mean is shifted based on the gradient of the upper bound, the agent isn't just "shaking" randomly; it is actively pushing into the unknown in directions that look promising.

Key Results

  • State-of-the-Art Efficiency: OAC achieved state-of-the-art sample efficiency on the Humanoid-v2 benchmark, reaching high performance significantly faster than SAC.
  • Superior Performance on Ant and Hopper: OAC consistently outperformed SAC and TD3 across most MuJoCo tasks, particularly in the early stages of learning where efficient exploration matters most.
OAC achieves state-of-the-art sample efficiency across various MuJoCo continuous control tasks compared to SAC, TD3, and DDPG.
  • Stability: Despite the more aggressive exploration strategy, OAC proved to be as stable as SAC, showing that the KL-constraint effectively manages the risks of optimism.
  • Ablation Success: The authors showed that the performance gains were specifically due to the bootstrapped uncertainty estimates; removing the "optimism" caused the performance to revert to standard SAC levels.

Limitations and Future Work

  • High Variance on Some Tasks: On the HalfCheetah environment, the results showed high variance, making it difficult to claim a definitive win over other algorithms in that specific domain.
  • Hyperparameter Sensitivity: OAC introduces new hyperparameters (like $\beta_{UB}$ for the level of optimism and $\delta$ for the KL-constraint). While the authors found stable defaults, these might require tuning for entirely new types of environments.
  • Simple Uncertainty Estimation: The current version uses a simple two-network bootstrap. The authors suggest that using more sophisticated uncertainty estimates (like larger ensembles or Bayesian neural networks) could further improve performance, albeit at a higher computational cost.

Summary updated

Abstract

Actor-critic methods, a type of model-free Reinforcement Learning, have been successfully applied to challenging tasks in continuous control, often achieving state-of-the art performance. However, wide-scale adoption of these methods in real-world domains is made difficult by their poor sample efficiency. We address this problem both theoretically and empirically. On the theoretical side, we identify two phenomena preventing efficient exploration in existing state-of-the-art algorithms such as Soft Actor Critic. First, combining a greedy actor update with a pessimistic estimate of the critic leads to the avoidance of actions that the agent does not know about, a phenomenon we call pessimistic underexploration. Second, current algorithms are directionally uninformed, sampling actions with equal probability in opposite directions from the current mean. This is wasteful, since we typically need actions taken along certain directions much more than others. To address both of these phenomena, we introduce a new algorithm, Optimistic Actor Critic, which approximates a lower and upper confidence bound on the state-action value function. This allows us to apply the principle of optimism in the face of uncertainty to perform directed exploration using the upper bound while still using the lower bound to avoid overestimation. We evaluate OAC in several challenging continuous control tasks, achieving state-of the art sample efficiency.
BibTeX
@inproceedings{ciosek2019better,
  author = {Kamil Ciosek and Quan Vuong and Robert Loftin and Katja Hofmann},
  title = {{Better Exploration with Optimistic Actor Critic}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2019},
  url = {https://openreview.net/forum?id=un-AbhNjiG},
}
Figure