Abstract
Modeling policies in reproducing kernel Hilbert space (RKHS) renders policy gradient reinforcement learning algorithms non-parametric. As a result, the policies become very flexible and have a rich representational potential without a pre-defined set of features. However, their performances might be either non-covariant under re-parameterization of the chosen kernel, or very sensitive to step-size selection. In this paper, we propose to use a general framework to derive a new RKHS policy search technique. The new derivation leads to both a natural RKHS actor-critic algorithm and a RKHS expectation maximization (EM) policy search algorithm. Further, we show that kernelization enables us to learn in partially observable (POMDP) tasks which is considered daunting for parametric approaches. Via sparsification, a small set of "support vectors" representing the history is shown to be effectively discovered. For evaluations, we use three simulated (PO)MDP reinforcement learning tasks, and a simulated PR2's robotic manipulation task. The results demonstrate the effectiveness of the new RKHS policy search framework in comparison to plain RKHS actor-critic, episodic natural actor-critic, plain actor-critic, and PoWER approaches.
Introduction
In traditional reinforcement learning, policies are typically "parametric," meaning they rely on a predefined set of features designed by human experts. This reliance creates a significant bottleneck: if the features are poorly designed, the agent's learning potential is fundamentally limited. Non-parametric methods using Reproducing Kernel Hilbert Space (RKHS) address this by allowing policies to be represented directly by the data, offering immense representational flexibility.
However, previous RKHS policy search methods suffered from two major flaws:
- Non-covariance: Changing kernel hyperparameters (like Gaussian width) could arbitrarily shift gradient directions, leading to erratic learning.
- Step-size Sensitivity: Standard gradient methods are notoriously sensitive to the learning rate ($\alpha$), where suboptimal choices lead to divergence or slow convergence.
Methodology
The authors propose a general framework for RKHS policy search based on lower-bound functional optimization, resulting in two key algorithms:
- RKHS Natural Actor-Critic (RKHS-NAC): By deriving the Fisher Information Metric specifically for the RKHS manifold, the update becomes covariant. This ensures that the learning path remains consistent even if the kernel parameterization changes.
- RKHS-PoWER: Inspired by the Expectation-Maximization (EM) algorithm, this version treats policy updates as a weighted regression problem, analytically calculating updates and bypassing the need for manual step-size tuning.
To prevent the policy from growing infinitely with data, the authors use sparsification (kernel matching pursuit) to represent the policy using a small set of "support vectors." For Partially Observable Markov Decision Processes (POMDPs), kernels are used to compare histories of observations, a task traditionally difficult for parametric models.
Experimental Evaluation
The algorithms were evaluated on four simulated benchmarks: a 1D navigation toy domain, a classic inverted pendulum (MDP), a partially observable inverted pendulum (POMDP), and a high-dimensional PR2 robotic manipulation task.
Key Results
- Robustness: In the toy domain, RKHS-NAC maintained near-identical performance across wildly different kernel bandwidths, whereas standard RKHS-AC failed when poorly tuned (see Figure above).
- Efficiency: In the Inverted Pendulum task, RKHS-NAC and RKHS-PoWER reached optimal returns significantly faster than standard Actor-Critic or eNAC.
- POMDP Superiority: On the partially observable pendulum, RKHS-NAC significantly outperformed the best parametric contender (eNAC), proving the non-parametric approach's ability to capture history-based nuances.
- Robotics: The PR2 door-opening task demonstrated that these methods scale to complex, high-dimensional robotic actions, with RKHS-PoWER showing the most stable learning curve.
Conclusion
This work demonstrates that moving policy search into RKHS provides a flexible and robust framework for reinforcement learning. By introducing natural gradients and EM-based updates, the proposed RKHS-NAC and RKHS-PoWER algorithms overcome the limitations of non-covariance and step-size sensitivity. These methods excel in complex tasks and partially observable environments where traditional parametric approaches struggle. Future research will explore real-world applications and further optimization of kernel selection.
Summary updated