TL;DR
This paper introduces Non-Monolithic Pre-training with Successor Features (NMPS), a novel approach to unsupervised pre-training for reinforcement learning agents. NMPS decouples exploration and exploitation into separate agents, improving task inference and generalization compared to existing monolithic methods.
What Problem Does This Paper Address?
Unsupervised pre-training in reinforcement learning (RL) aims to equip agents with generalizable skills without explicit rewards, enabling faster adaptation to new tasks. Current methods often struggle with local optima due to a unified approach to exploration and exploitation. Specifically, approaches relying on successor features (SFs) to decouple environment dynamics from rewards, while promising for task adaptation, can suffer from inefficient exploration and limited skill space generalization. Existing methods like APS, which combine intrinsic rewards for exploration and exploitation, can lead to interference and suboptimal performance. This paper addresses the challenge of effectively separating exploration and exploitation within an SF-based pre-training framework to achieve better generalization and avoid these limitations.
Methods
The core innovation of this paper is the Non-Monolithic Exploration Methodology. Instead of a single agent handling both exploration and exploitation, NMPS employs two distinct agents: an "Exploit" agent focused on task inference and an "Explor" agent dedicated to exploration. These agents are coordinated by a "mode-switching controller" (termed "Homeo") that decides when to switch between exploration and exploitation modes.
Key aspects of the methodology include:
- Successor Features (SFs): The agents leverage SFs to represent environment dynamics independently of rewards, facilitating transfer learning.
- Decoupled Intrinsic Rewards: The unified intrinsic reward of previous methods (like APS) is split. The Exploit agent uses its intrinsic reward for exploitation, while the Explor agent uses its intrinsic reward for exploration. This separation aims to prevent interference.
- Separate Agents: The Exploit agent is responsible for "task inference" and is used for fine-tuning. The Explor agent is primarily for exploration during pre-training and is discarded afterward.
- Mode-Switching Controller (Homeo): This controller uses the "value promise discrepancy" (a measure of value function difference over time) to decide which agent's action to take.
- Competence-Based Exploration: The Explor agent can leverage competence-based algorithms (like DIAYN) to enhance exploration flexibility and generalization, particularly for the Exploit agent's discriminator.
- Optimized Training: The paper proposes an optimized training method for these separate agents, allowing them to use shared or separate replay buffers depending on the specific NMPS variant.
Datasets
The experiments were conducted in three domains from the DeepMind Control Suite (DMC): Walker, Jaco Arm, and Quadruped. These are standard continuous control tasks used for evaluating RL algorithms. No new benchmarks were introduced.
Key Insights and Ideas
- Separation is Key: Decoupling exploration and exploitation into distinct agents, rather than a single monolithic agent, significantly improves performance. This allows each agent to focus on its core task without interference.
- SFs Enable Transfer: Successor Features are crucial for enabling agents to adapt to new tasks by decoupling environment dynamics from rewards.
- Homeo's Role: The mode-switching controller (Homeo) plays a critical role in coordinating the agents and influencing the exploration-exploitation balance. Its target rate parameter is a key factor in performance.
- Competence-Based Exploration Boosts Generalization: Using competence-based algorithms for the exploration agent can enhance the generalization capabilities of the exploitation agent's discriminator, helping to overcome limitations in small skill spaces.
- Task Inference vs. Exploration: The paper highlights the importance of dedicating one agent to "task inference" (Exploit) and another to "exploration" (Explor) for effective pre-training.
Key Results
NMPS consistently outperformed APS, a strong baseline, across all three evaluated domains (Walker, Jaco Arm, and Quadruped).
- Walker: NMPS_X_exploite* showed overwhelming performance, significantly improving upon APS.
- Jaco Arm: NMPS_X_exploitex demonstrated better performance than APS after approximately 0.25M steps.
- Quadruped: NMPS_D_sepex_D_A10 achieved the best performance after around 0.68M steps, outperforming APS and other variants.
- Overall: NMPS variants generally showed superior performance to APS, with specific NMPS configurations excelling in different domains. The paper also compared NMPS against DIAYN and SMM, showing competitive or superior results.
Limitations and Future Work
- Explor Agent Discarded: The Explor agent is discarded after pre-training. Future work could explore scenarios where the Explor agent's learned skills are retained or utilized in some capacity.
- Agent Type Dependence: The performance of NMPS variants can depend on the domain, suggesting that the optimal configuration might require domain-specific tuning.
- Feature/Skill Training: The paper notes that the "train of feature or skill" factor for the Explor agent can significantly impact performance and is domain-dependent. Further investigation into optimal feature/skill training strategies could be beneficial.
- Exploration Beyond Features/Skills: The authors suggest that future research could explore models for agents that are not based on features or skills for the Explor agent.
Summary updated