Abstract
We present a reinforcement learning algorithm for learning sparse non-parametric controllers in a Reproducing Kernel Hilbert Space. We improve the sample complexity of this approach by imposing a structure of the state-action function through a normalized advantage function (NAF). This representation of the policy enables efficiently composing multiple learned models without additional training samples or interaction with the environment. We demonstrate the performance of this algorithm on learning obstacle-avoidance policies in multiple simulations of a robot equipped with a laser scanner while navigating in a 2D environment. We apply the composition operation to various policy combinations and test them to show that the composed policies retain the performance of their components. We also transfer the composed policy directly to a physical platform operating in an arena with obstacles in order to demonstrate a degree of generalization.
Introduction
In traditional deep reinforcement learning, models are often "monolithic." If two robots are trained in different environments—for instance, one in a circular room and another in narrow corridors—combining their neural network weights is nearly impossible. Merging specialized weights typically results in "catastrophic forgetting" or requires extensive retraining.
This is a major hurdle for distributed robot teams that may only briefly reconnect to share information. This paper addresses the need for composable learning: a method to join models learned by different agents into a single model that retains the strengths of all components without needing new data or further interaction with the environment.
Methods: Kernelized Advantage and Composition
The authors move away from deep neural networks, instead utilizing Reproducing Kernel Hilbert Space (RKHS) representations to maintain modularity.
- KNAF (Kernel Normalized Advantage Functions): To handle continuous actions like steering, the authors kernelize the NAF algorithm. This represents the policy as a weighted sum of "kernel evaluations" at specific training states, known as the "support."
- Sparsification with KOMP: To prevent the model from growing indefinitely, Kernel Orthogonal Matching Pursuit (KOMP) is used to keep the dictionary of important states small and computationally efficient.
- Density-Based Composition: When merging multiple policies, the algorithm evaluates the kernel density of the support points. If a specific policy has many support points in a region of the state space, it is treated as the "local expert" for that region.
Experimental Setup
The researchers evaluated their approach using four distinct simulated 2D maps and a physical robot platform:
- Simulated Environments: The maps included Round (simple symmetry), Circuit 1 (tight turns), Circuit 2 (left/right hallways), and Maze (a complex combination of features).
- Physical Robot: A Scarab robot equipped with a Hokuyo laser scanner was tested in a laboratory arena with physical obstacles to verify simulation-to-reality transfer.
Key Results and Insights
- Support as Reliability: In kernel methods, the "support" (the specific data points the model remembers) directly represents where the model is applicable. This makes merging models as intuitive as merging two sets of points on a map.
- Zero-Shot Generalization: The composed policy (combining all four training environments) outperformed individual policies on almost every map. For example, a policy trained only on the "Round" environment failed miserably in the "Maze" (reward of -11,663), while the composed policy achieved a perfect reward of 1000.
- Real-World Success: When transferred to a physical Scarab robot, the policy trained only on the "Round" environment crashed 3 times. The composed policy had 0 crashes and successfully navigated multiple loops of the physical arena.
Conclusion
The primary limitation of this approach is the curse of dimensionality. While kernel methods work well for 2D navigation with low-dimensional laser inputs, they can struggle with high-dimensional data like raw camera images. The authors suggest that future work could involve using auto-encoders to compress high-dimensional data into a lower-dimensional feature space before applying these sparse kernel methods, enabling multi-robot systems to actively exchange "knowledge packets" across diverse environments.
Summary updated