Abstract
This paper introduces Cross-learning, a novel framework for Multi-Task Reinforcement Learning (MTRL) designed to operate within Reproducing Kernel Hilbert Spaces (RKHS). By leveraging the mathematical properties of RKHS, the authors enable multiple agents to learn related tasks collaboratively without sacrificing individual task specialization. The approach utilizes a central "meta-policy" that acts as an anchor for task-specific agents, facilitating knowledge transfer and improving generalization to unseen environments while maintaining computational efficiency through relaxed projections and kernel pruning.
Introduction
Standard Reinforcement Learning (RL) is often limited by high sample complexity and a lack of generalization across varying environments. Multi-task learning (MTL) addresses these issues by sharing information across related tasks. However, existing MTL methods typically fall into two extremes: fully distributed learning, which lacks knowledge sharing, and strict consensus, which forces all agents to adopt an identical policy.
The authors propose Cross-learning as a "middle ground." By formulating the problem as a constrained optimization in RKHS, agents are allowed to specialize in their specific tasks while being constrained to stay within a defined distance of a central, shared policy. This structure ensures that agents benefit from collective knowledge without being overly restricted by it.
Methods
The core of the Cross-learning framework involves three key technical components:
- RKHS Policy Representation: Policies are represented as elements of an RKHS, allowing for a non-parametric approach where the distance between policies is mathematically well-defined and continuous.
- The Cross-Learning Constraint: Each task-specific policy $h_i$ is constrained to remain within an $\epsilon$-distance of a central policy $g$. This $\epsilon$ acts as a "bargaining parameter" that balances the trade-off between individual task performance and collective consensus.
- Relaxed Projection and CDKOMP: To avoid the high computational cost of strict projections, the authors introduce a "relaxed" version with a closed-form solution. Additionally, they employ Common Dictionary Kernel Orthogonal Matching Pursuit (CDKOMP) to prune less informative kernels, preventing the model size from growing uncontrollably over time.
The iterative implementation of these steps is detailed in the Cross-learning algorithm, which alternates between policy gradient updates, projection, and dictionary reduction.
Results and Discussion
The framework was evaluated using a simulated continuous navigation environment where a robot must reach a goal while avoiding obstacles of different sizes.
- Superior Generalization: The central policy $g$ effectively learned the abstract concept of obstacle avoidance. In tests involving an unseen elliptical obstacle, the cross-learned policy outperformed both agnostic (single-task) agents and strict consensus policies, navigating more efficiently without crashing.
- Stability and Efficiency: The relaxed projection method provided a computationally efficient alternative to strict quadratic programming while maintaining high performance. Furthermore, the shared knowledge across tasks acted as a stabilizer, resulting in lower variance in costs during the later stages of training compared to independent agents.
Conclusion
Cross-learning in RKHS offers a mathematically rigorous and efficient solution for multi-task reinforcement learning. By anchoring individual agents to a central policy, the framework achieves a balance between specialization and generalization. Future work may involve dynamic adjustment of the $\epsilon$ constraint to automatically account for varying degrees of task relatedness.
Summary updated