Abstract
In robotics and control, sample efficiency is a critical metric for evaluating reinforcement learning algorithms. This study compares the performance of Deep Deterministic Policy Gradient (DDPG), an actor-critic method, against Covariance Matrix Adaptation Evolution Strategy (CMA-ES), a derivative-free direct policy search method. Using the Continuous Mountain Car benchmark, we demonstrate that DDPG's utilization of analytic gradients and experience replay leads to superior sample efficiency, consistency, and scalability compared to the black-box optimization approach of CMA-ES.
Introduction
Sample efficiency—the ability to learn an optimal policy with minimal environment interactions—is paramount in robotics due to the high costs associated with physical wear, energy consumption, and time. Historically, the robotics community favored direct policy search methods like CMA-ES for their perceived robustness in low-dimensional spaces. However, the advent of Deep Reinforcement Learning (Deep RL) has introduced sophisticated actor-critic architectures. This paper investigates whether modern Deep RL algorithms, specifically DDPG, have surpassed traditional black-box methods in the metrics most relevant to practical robotics.
Methodology
We compare two distinct optimization paradigms:
- Deep Deterministic Policy Gradient (DDPG): An off-policy actor-critic algorithm. It employs an Actor network to determine actions and a Critic network to estimate the value of those actions. DDPG integrates techniques such as a replay buffer for experience reuse and target networks to stabilize the learning objective.
- Covariance Matrix Adaptation Evolution Strategy (CMA-ES): A stochastic, gradient-free numerical optimization method. It iteratively updates a search distribution (a multivariate normal distribution) based on the performance of a population of candidate solutions.
To ensure a controlled comparison, both algorithms utilized identical neural network architectures for their controllers. Batch normalization was omitted in DDPG to maintain structural parity with the CMA-ES controllers.
Experimental Setup
The algorithms were evaluated on the Continuous Mountain Car benchmark. This environment presents a classic sparse-reward challenge: a low-powered vehicle must oscillate within a valley to accumulate sufficient momentum to reach a hilltop. The sparsity of the reward signal makes exploration difficult, providing a rigorous test for sample efficiency.
Key Insights
The observed performance gap between DDPG and CMA-ES can be attributed to three primary factors:
- Analytic Gradients: DDPG leverages backpropagation to compute exact parameter updates. In contrast, CMA-ES must sample a population to estimate a search direction, which is inherently less efficient.
- Experience Replay: DDPG’s off-policy nature allows it to store and repeatedly learn from past transitions. CMA-ES is typically local; once a generation is evaluated, the associated samples are discarded.
- The Critic as an Informant: The Critic network provides a dense signal for the Actor. Once the Critic begins to approximate the value function, the Actor can refine its policy without requiring additional environment steps for every update.
Results
Experimental results across multiple trials confirm that DDPG significantly outperforms CMA-ES in several key dimensions:
- Convergence Speed: DDPG reached the target performance threshold in a fraction of the environment steps required by CMA-ES.
- Stability and Variance: DDPG exhibited lower variance across independent runs, whereas CMA-ES showed higher sensitivity to initial conditions.
- Scalability: As the number of network parameters increased (from 51 to 281), DDPG maintained stable performance, while CMA-ES suffered from slower convergence and increased instability in the higher-dimensional search space.
Conclusion
While CMA-ES remains a robust choice for certain low-dimensional or non-differentiable problems, DDPG demonstrates a clear advantage in sample efficiency for continuous control tasks. Future work should investigate whether these advantages persist when using specialized representations like Dynamic Movement Primitives (DMPs) and how DDPG's known sensitivity to hyperparameters can be mitigated to make it as "plug-and-play" as traditional evolution strategies.
Summary updated