Abstract
Direct policy search is a fundamental approach in reinforcement learning (RL), where policy parameters are optimized to maximize cumulative rewards. However, standard derivative-free optimization (DFO) methods often operate in batch modes that do not align with the sequential nature of RL evaluations. This paper presents SRACOS (Sequential RACOS), a classification-based optimization algorithm designed for sequential sampling. By updating its search model after every evaluation and reusing historical data, SRACOS achieves faster convergence and better scalability. Evaluations on synthetic benchmarks and high-dimensional control tasks, including OpenAI Gym and simulated helicopter hovering, show that SRACOS consistently outperforms existing DFO methods.
Introduction
In reinforcement learning, direct policy search treats the policy as a black box. Optimizing these policies—often high-dimensional neural networks—requires efficient derivative-free optimization. Traditional DFO methods, such as CMA-ES or the original RACOS, typically employ batch-mode sampling. In this paradigm, a set of solutions is sampled and evaluated before the model is updated. This is inefficient for RL tasks where evaluations occur sequentially (e.g., one robot trial at a time). Batch-mode algorithms "waste" evaluations by sampling blindly within a batch, failing to incorporate information from the most recent trials immediately.
Methods
SRACOS builds upon the RACOS framework, which uses classification to learn a promising search region (an axis-parallel box) containing good solutions while excluding poor ones. SRACOS introduces three critical enhancements for sequential optimization:
- Sequential Model Updates: Unlike batch methods, SRACOS updates its classification model immediately after each evaluation.
- Historical Data Reuse: The algorithm maintains a "Positive set" ($B^+$) of the best solutions found so far and a "Negative set" ($B^-$) of inferior solutions. Each new sample refines these sets and the resulting search region.
- Negative Set Replacement Strategies: To maintain the quality of the negative set, the authors proposed three strategies:
- Worst Replacement (WR): Replaces the worst-performing solution in $B^-$.
- Random Replacement (RR): Replaces a random solution in $B^-$.
- Largest Margin (LM): Replaces the solution closest to the positive set to preserve diversity.
Results
The performance of SRACOS was validated across synthetic functions and complex control environments.
Synthetic Benchmarks and Scalability
On synthetic functions like the convex Sphere and non-convex Ackley functions, SRACOS demonstrated superior convergence rates compared to batch-mode RACOS and other DFO baselines. As shown in the figures, the sequential update mechanism allows SRACOS to reach lower function values with significantly fewer evaluations. Notably, the choice of replacement strategy (WR, RR, or LM) had a negligible impact on performance, suggesting that the sequential update itself is the primary driver of efficiency.
Control Tasks: Helicopter Hovering and OpenAI Gym
In the challenging helicopter hovering task, SRACOS achieved a higher success rate (4/15) and higher average rewards than RACOS (2/15) and other methods. In the OpenAI Gym suite, SRACOS dominated across all eight tested tasks, including high-dimensional locomotion problems like HalfCheetah and Humanoid. For instance, in HalfCheetah, SRACOS reached a reward of approximately 36,719, significantly surpassing the 27,961 achieved by RACOS.
Conclusion
SRACOS effectively bridges the gap between classification-based optimization and the sequential requirements of direct policy search. By learning from every evaluation in real-time, it offers a scalable and efficient alternative to traditional batch-mode DFO. While the computational cost per update is higher, the reduction in required environment evaluations makes it highly suitable for RL applications where trials are time-consuming. Future work will explore more complex classification models and methods to improve policy robustness.
Summary updated