Actor-critic versus direct policy search: a comparison based on sample complexity

Arnaud de Froissard de Broissia, Olivier SigaudarXiv 2016 · arXiv

Abstract

In robotics and control, sample efficiency is a critical metric for evaluating reinforcement learning algorithms. This study compares the performance of Deep Deterministic Policy Gradient (DDPG), an actor-critic method, against Covariance Matrix Adaptation Evolution Strategy (CMA-ES), a derivative-free direct policy search method. Using the Continuous Mountain Car benchmark, we demonstrate that DDPG's utilization of analytic gradients and experience replay leads to superior sample efficiency, consistency, and scalability compared to the black-box optimization approach of CMA-ES.

Introduction

Sample efficiency—the ability to learn an optimal policy with minimal environment interactions—is paramount in robotics due to the high costs associated with physical wear, energy consumption, and time. Historically, the robotics community favored direct policy search methods like CMA-ES for their perceived robustness in low-dimensional spaces. However, the advent of Deep Reinforcement Learning (Deep RL) has introduced sophisticated actor-critic architectures. This paper investigates whether modern Deep RL algorithms, specifically DDPG, have surpassed traditional black-box methods in the metrics most relevant to practical robotics.

Methodology

We compare two distinct optimization paradigms:

  1. Deep Deterministic Policy Gradient (DDPG): An off-policy actor-critic algorithm. It employs an Actor network to determine actions and a Critic network to estimate the value of those actions. DDPG integrates techniques such as a replay buffer for experience reuse and target networks to stabilize the learning objective.
  2. Covariance Matrix Adaptation Evolution Strategy (CMA-ES): A stochastic, gradient-free numerical optimization method. It iteratively updates a search distribution (a multivariate normal distribution) based on the performance of a population of candidate solutions.

To ensure a controlled comparison, both algorithms utilized identical neural network architectures for their controllers. Batch normalization was omitted in DDPG to maintain structural parity with the CMA-ES controllers.

Visualizations of the learned control landscapes for the Continuous Mountain Car task. These heatmaps illustrate the internal representations developed by the algorithms to navigate the state space.

Experimental Setup

The algorithms were evaluated on the Continuous Mountain Car benchmark. This environment presents a classic sparse-reward challenge: a low-powered vehicle must oscillate within a valley to accumulate sufficient momentum to reach a hilltop. The sparsity of the reward signal makes exploration difficult, providing a rigorous test for sample efficiency.

Key Insights

The observed performance gap between DDPG and CMA-ES can be attributed to three primary factors:

  • Analytic Gradients: DDPG leverages backpropagation to compute exact parameter updates. In contrast, CMA-ES must sample a population to estimate a search direction, which is inherently less efficient.
  • Experience Replay: DDPG’s off-policy nature allows it to store and repeatedly learn from past transitions. CMA-ES is typically local; once a generation is evaluated, the associated samples are discarded.
  • The Critic as an Informant: The Critic network provides a dense signal for the Actor. Once the Critic begins to approximate the value function, the Actor can refine its policy without requiring additional environment steps for every update.

Results

Experimental results across multiple trials confirm that DDPG significantly outperforms CMA-ES in several key dimensions:

  • Convergence Speed: DDPG reached the target performance threshold in a fraction of the environment steps required by CMA-ES.
  • Stability and Variance: DDPG exhibited lower variance across independent runs, whereas CMA-ES showed higher sensitivity to initial conditions.
  • Scalability: As the number of network parameters increased (from 51 to 281), DDPG maintained stable performance, while CMA-ES suffered from slower convergence and increased instability in the higher-dimensional search space.
Performance comparison between DDPG (blue) and CMA-ES (green). (a) Total environment steps required to reach the goal. (b) Average reward per episode over time. DDPG demonstrates vastly superior sample efficiency and faster convergence.

Conclusion

While CMA-ES remains a robust choice for certain low-dimensional or non-differentiable problems, DDPG demonstrates a clear advantage in sample efficiency for continuous control tasks. Future work should investigate whether these advantages persist when using specialized representations like Dynamic Movement Primitives (DMPs) and how DDPG's known sensitivity to hyperparameters can be mitigated to make it as "plug-and-play" as traditional evolution strategies.

Summary updated

Abstract

Sample efficiency is a critical property when optimizing policy parameters for the controller of a robot. In this paper, we evaluate two state-of-the-art policy optimization algorithms. One is a recent deep reinforcement learning method based on an actor-critic algorithm, Deep Deterministic Policy Gradient (DDPG), that has been shown to perform well on various control benchmarks. The other one is a direct policy search method, Covariance Matrix Adaptation Evolution Strategy (CMA-ES), a black-box optimization method that is widely used for robot learning. The algorithms are evaluated on a continuous version of the mountain car benchmark problem, so as to compare their sample complexity. From a preliminary analysis, we expect DDPG to be more sample efficient than CMA-ES, which is confirmed by our experimental results.
BibTeX
@misc{broissia2016actor,
  author = {Arnaud de Froissard de Broissia and Olivier Sigaud},
  title = {{Actor-critic versus direct policy search: a comparison based on sample complexity}},
  year = {2016},
  eprint = {1606.09152},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1606.09152},
}
Figure