Abstract
SEAPoT-RL is a reinforcement learning (RL) framework designed to facilitate rapid adaptation to environmental changes by identifying specific affected sub-spaces. By employing Bayesian change point detection and selective exploration, the algorithm maximizes the reuse of prior knowledge, achieving a significant "jump start" in performance and faster convergence compared to existing transfer learning methods.
Introduction
Transfer learning in RL aims to leverage knowledge from a source task to accelerate learning in a target task. However, standard agents often struggle with environmental shifts, either failing to adapt or requiring exhaustive re-exploration. SEAPoT-RL addresses this by autonomously detecting changes and performing "surgical" exploration only where necessary, avoiding the need for human-provided hints or global re-learning.
Methods
The SEAPoT-RL framework operates in two primary phases to handle environmental transitions efficiently:
- Change Detection: Using Bayesian change point detection, the agent monitors sensor observations as a time series. Significant deviations from expected values trigger a "change point," signaling environmental modifications such as the addition or removal of obstacles.
- Selective Exploration via Sub-space Extraction: Upon detection, the agent extracts a local sub-space (MDP $M'$) using an $n$-step closure around the change point. This local MDP is solved using the Factored-RMAX algorithm. The final policy is a composition: the agent uses the newly learned local policy within the sub-space and reverts to the original source policy upon reaching the boundary where the environment matches the source task.
Experimental Evaluation
The algorithm was evaluated in a simulated 10x10 grid world (8,000 states) representing an assistive care home. The task involved picking up and dropping off objects with stochastic transitions (80% success rate). SEAPoT-RL was compared against a "no transfer" baseline and three state-of-the-art methods: Probabilistic Policy Reuse (PPR), Reward Shaping (RS), and Bayesian Policy Reuse (BPR).
Results
SEAPoT-RL demonstrated a superior "jump start," beginning with an average reward of approximately -400, compared to -2000 for the agent starting from scratch. As shown in the performance comparison, SEAPoT-RL maintains a higher cumulative average reward and reaches near-optimal performance significantly faster than PPR, RS, and BPR.
Conclusion
SEAPoT-RL provides an efficient, autonomous solution for policy transfer by isolating exploration to changed sub-spaces. By avoiding catastrophic forgetting and redundant global exploration, it offers a robust framework for agents operating in dynamic environments. Future work will address limitations regarding sub-space size optimization and the development of more proactive exploration strategies.
Summary updated