TL;DR
This paper investigates the performance of on-policy reinforcement learning algorithms, specifically Proximal Policy Optimization (PPO), in continual learning scenarios. The authors find that while PPO struggles with average performance compared to off-policy methods, it shows promising results in terms of forward transfer and forgetting, suggesting a need for tailored on-policy continual learning approaches.
What Problem Does This Paper Address?
Many real-world environments, especially in robotics, are dynamic and change over time. Continual Reinforcement Learning (CRL) aims to train agents that can adapt to these changing environments without forgetting previously learned skills. However, most existing CRL research focuses on off-policy algorithms, which can leverage replay buffers to store and reuse past experiences. This is problematic for on-policy algorithms like PPO, which learn directly from the data generated by their current policy and cannot easily benefit from such replay buffers. This paper bridges this gap by evaluating PPO's effectiveness in a standard CRL benchmark, exploring its limitations and potential.
Methods
The core of this study is the evaluation of Proximal Policy Optimization (PPO), a popular on-policy reinforcement learning algorithm, within a continual learning framework. The authors compare PPO's performance against an off-policy algorithm, Soft Actor-Critic (SAC), which is the standard in the Continual World benchmark. They also incorporate three common continual learning methods:
- Fine-tuning: A baseline where the agent's parameters are updated on new tasks without explicit mechanisms to prevent forgetting.
- L2 Regularization: A method that penalizes large weights, aiming to keep the model parameters close to their previous values.
- Elastic Weight Consolidation (EWC): A technique that identifies and protects important weights for previously learned tasks.
- PackNet: An approach that involves iterative pruning to add new tasks to a network while preserving performance on old ones.
The evaluation focuses on metrics relevant to CRL, including average performance, forward transfer (how well knowledge from previous tasks helps new ones), and forgetting (how much performance degrades on old tasks after learning new ones).
Datasets
The experiments are conducted using the Continual World robotic benchmark. This benchmark is designed to evaluate RL agents in continual learning environments. It presents a sequence of tasks, each with a specific interval for training, and evaluates performance throughout the learning process. The authors use a sequence of N=5 tasks for their experiments.
Key Insights and Ideas
- On-policy vs. Off-policy in CRL: The study highlights fundamental differences in how on-policy and off-policy algorithms behave in continual learning. Off-policy methods, with their replay buffers, seem to have an inherent advantage in maintaining performance across tasks.
- PPO's Performance Trade-offs: While PPO generally achieves lower average performance than SAC in this CRL setting, it demonstrates competitive results in terms of forward transfer and forgetting. This suggests that on-policy methods might not be inherently worse but require different adaptation strategies.
- The Role of Replay Buffers: The authors hypothesize that the superior performance of off-policy methods, particularly with techniques like PackNet, is significantly due to the replay buffer. This buffer allows for more robust retraining and consolidation of knowledge.
- Forward Transfer Potential: Interestingly, PPO, when combined with CL methods, shows higher forward transfer than SAC in some instances, indicating its potential to leverage prior knowledge effectively.
- Need for Tailored Methods: The findings strongly suggest that existing CRL methods, often designed with off-policy learning in mind, may not be optimal for on-policy algorithms. This points to a research gap in developing CRL techniques specifically for on-policy agents.
Key Results
- Average Performance: PPO consistently resulted in markedly inferior average performance compared to SAC across all tested CL methods. For example, in fine-tuning, PPO achieved 0.164 performance compared to SAC's 0.212. With PackNet, PPO's performance was 0.387, while SAC's was 0.842.
- Forward Transfer: PPO showed promising forward transfer, with values like 0.271 for fine-tuning (vs. SAC's 0.764) and 0.073 for PackNet (vs. SAC's -0.011). Notably, PPO with fine-tuning did not exhibit any forward transfer, but with other CL methods, it was higher than SAC.
- Forgetting: PPO exhibited some degree of forgetting, particularly with PackNet (0.073 forgetting), whereas PackNet with SAC showed no signs of forgetting (-0.011). However, PPO's forgetting was generally less severe than SAC's in some scenarios (e.g., fine-tuning: PPO 0.271 vs. SAC 0.764).
Limitations and Future Work
The authors acknowledge that their findings suggest current on-policy methods are suboptimal in terms of average performance. They hypothesize that this is partly due to the reliance of off-policy algorithms on replay buffers. A key limitation is that the study primarily uses PPO and doesn't explore a broader range of on-policy algorithms.
Future work will focus on identifying the precise reasons for the observed discrepancies between on-policy and off-policy algorithms in CRL. The authors intend to develop CL methods specifically tailored to address the shortcomings of on-policy algorithms, aiming to improve their performance and robustness in dynamic environments.
Summary updated