Abstract
In unsupervised reinforcement learning, agents aim to acquire a library of behaviors in the absence of extrinsic rewards to facilitate rapid adaptation to downstream tasks. Successor Features (SFs) are a powerful tool for this purpose, as they mathematically decouple environment dynamics from reward functions. However, current state-of-the-art methods like Active Pre-training with Successor Features (APS) employ a monolithic approach, using a single reward function that conflates exploration (discovering new states) and exploitation (learning to distinguish skills). This interference often leads to inefficient exploration and violates the assumptions required for SFs to function optimally. We propose NMPS (Non-Monolithic unsupervised Pre-training with Successor features), which decouples these objectives into two separate agents managed by a homeostasis controller, leading to superior performance across complex continuous control tasks.
The Interference Problem in Monolithic RL
Traditional unsupervised RL methods attempt to maximize a "unified" intrinsic reward. This reward typically combines an entropy-maximizing term for exploration and a discriminative term for skill learning. The authors argue that this combination creates a fundamental conflict. Because the exploration signal is often high-variance and non-stationary, it "pollutes" the learning signal for the Successor Features. This violation of linear regression assumptions prevents the agent from accurately mapping the environment's dynamics, often causing the model to get stuck in local optima or fail to explore the state space comprehensively.
The NMPS Framework
NMPS moves away from the monolithic architecture in favor of a non-monolithic, dual-agent system. The framework consists of three primary components:
- The Exploit Agent: This agent focuses on task inference and Successor Feature learning. It is responsible for refining the agent's understanding of the environment's features and how they relate to potential rewards.
- The Explor Agent: This agent is dedicated solely to maximizing entropy and visiting novel states. By isolating this behavior, the agent can engage in "pure" exploration without the pressure of maintaining stable feature representations.
- The Homeostasis Controller (Homeo): This is the "brain" of the system that decides when to switch between the two agents. It uses a metric called Value Promise Discrepancy—the difference between the expected value of a state and the actual cumulative reward received over a specific window. If the Exploit agent is consistently failing to meet its value expectations, the controller triggers the Explor agent to find new, more informative states.
This system is built upon Deep Deterministic Policy Gradient (DDPG) to effectively manage continuous action spaces.
Experimental Evaluation
The model was evaluated using the DeepMind Control Suite (DMC), focusing on three distinct domains of increasing complexity:
- Walker: A bipedal robot learning locomotion.
- Jaco Arm: A robotic arm performing manipulation tasks.
- Quadruped: A four-legged robot navigating complex movement.
Key Results
Across all tested environments, NMPS demonstrated significant improvements over the baseline APS and other competence-based methods like DIAYN and SMM.
- Walker Task: The NMPS variants showed overwhelming performance, significantly outperforming APS across all training steps. This suggests that the decoupling of rewards allows for much more efficient locomotion learning.
- Jaco Arm: While NMPS initially experienced a slight overhead due to the mode-switching mechanism, it surpassed APS after approximately 0.25M steps and maintained a higher performance ceiling throughout the remainder of the training.
- Quadruped: This task highlighted the modularity of NMPS. By using DIAYN as the "Explor" component within the NMPS framework, the agent was able to overcome the "small skill space" limitations that usually prevent competence-based algorithms from succeeding in high-dimensional environments.
Discussion: "When" vs. "How" to Explore
A critical insight of this research is the shift in focus from how to explore to when to explore. While traditional RL research focuses on exploration strategies (e.g., noise injection or curiosity), NMPS demonstrates that the timing of exploration is equally vital. By switching entirely to a dedicated exploration policy when the exploitation agent becomes "confused" (high value promise discrepancy), the system avoids the diluted signals inherent in combined reward functions.
Furthermore, the use of a shared replay buffer allows the Exploit agent to learn from the diverse states discovered by the Explor agent. This ensures that the Successor Features are trained on a rich variety of data without the Exploit agent having to perform the high-entropy, "messy" exploration itself.
Limitations and Future Work
Despite its success, NMPS introduces additional complexity. The "target rate" ($ ho$) for the Homeostasis controller is a sensitive hyperparameter that requires tuning for different environments. Additionally, managing two separate agents increases the computational footprint of the pre-training phase. Future research may explore more generalized curiosity-driven mechanisms for the "Explor" agent that do not rely strictly on predefined features or skills.
Summary updated