Introduction
In Reinforcement Learning (RL), agents typically begin as a "blank slate," necessitating extensive exploration to learn complex tasks. Transfer Learning (TL) offers a solution by enabling agents to leverage knowledge from a "source" task to accelerate learning in a "target" task. A formal framework for this is MDP Homomorphisms, which mathematically define task similarities. However, traditional homomorphisms are often too rigid, requiring exact reward matching, and are computationally expensive to discover, particularly in continuous, high-dimensional spaces.
Methodology: Factored State Variable Mappings
This paper introduces a scalable approach to transfer learning by mapping state variables rather than individual states. This "variable-to-variable" mapping allows the use of MDP Homomorphisms in continuous domains like Robosoccer.
The authors propose three primary innovations to overcome the limitations of traditional transfer methods:
- Factored Mappings: Instead of mapping every possible state $s$ to $s'$, they map compact variables (e.g., "distance to teammate"). This bypasses the "curse of dimensionality" in continuous spaces.
- Relaxed Reward Constraints: By ignoring strict reward-matching, the agent can reuse behaviors that serve as useful "subgoals" even if the ultimate objectives differ.
- Transfer as Options: Transferred policies are treated as "Options" (macro-actions). Using Intra-option Q-learning, the agent learns when to employ these borrowed skills while simultaneously developing new actions. If a skill proves useless, the agent eventually learns to ignore it.
Experimental Evaluation
The approach was evaluated on two distinct benchmarks:
- Blocksworld: A discrete domain involving block stacking. Policies were transferred from 3-6 block environments to a 7-block target.
- Robosoccer Keepaway: A complex, continuous multi-agent simulator where "keepers" must maintain possession. Knowledge was transferred from a 3v2 scenario to a more challenging 4v3 scenario.
Results and Discussion
The results demonstrate significant improvements in learning efficiency. In the Robosoccer Keepaway task, agents utilizing transfer reached target performance levels (holding the ball for 8 seconds) approximately 30 minutes faster than those learning from scratch.
Key insights from the experiments include:
- Bootstrapping: Transfer provides a significant head start without compromising long-term performance. The agent eventually matches the performance of the "from-scratch" learner but reaches it much sooner.
- Distraction Risk: In Blocksworld, transferring from overly simple sources (e.g., 3 blocks to 7) initially hindered learning. These simple policies were "distracting" because they were applicable in many states but lacked the complexity needed for the target goal.
- Scalability: The method effectively bridges the gap between different numbers of agents in multi-agent systems, demonstrating that variable-based homomorphisms can handle structural changes in the environment.
Conclusion
By focusing on state variable mappings and treating transferred knowledge as options, this work makes MDP Homomorphisms practical for continuous RL. While the current mappings are manually defined, this framework provides a robust foundation for future research into automated mapping discovery and symmetry exploitation within complex domains.
Summary updated