Abstract
Unsupervised Reinforcement Learning (URL) typically follows a two-stage paradigm of task-agnostic pretraining followed by task-specific finetuning. However, this approach is often hampered by non-stationarity in intrinsic rewards and catastrophic forgetting of exploratory behaviors during finetuning. We introduce Self-Reference (SR), an architectural augmentation that allows agents to explicitly retrieve and aggregate their own historical experiences. By providing temporal context, SR stabilizes the learning process and preserves robust exploration strategies. Our evaluations on the Unsupervised RL Benchmark (URLB) demonstrate that SR achieves state-of-the-art performance, improving the Interquartile Mean (IQM) by up to 17% while allowing for zero-overhead deployment via policy distillation.
Introduction
In Unsupervised Reinforcement Learning, agents are initially tasked with exploring an environment without external rewards, often driven by intrinsic motivations like novelty or surprise. While effective for discovery, this leads to a significant challenge: non-stationarity. As an agent explores, previously "novel" states become familiar, causing the intrinsic reward function to shift constantly. Most current algorithms fail to account for this history, leading to unstable training. Furthermore, during the transition to a downstream task (finetuning), agents often suffer from catastrophic forgetting, where they abandon the diverse behaviors learned during pretraining in favor of narrow, sub-optimal strategies for the specific task.
Methodology: The Self-Reference Module
The Self-Reference (SR) module addresses these challenges by enabling the agent to "look back" at its history. The architecture consists of three core components integrated into the standard RL loop:
- Query Module: A neural network trained via Proximal Policy Optimization (PPO) that generates a query based on the current observation. This module learns to dynamically seek historical information—such as previous visits to similar states or potential future trajectories—that is most relevant to the current context.
- Retrieval Mechanism: Utilizing the Faiss library for efficient similarity search, the agent retrieves the top k experience sequences (short trajectories) from its replay buffer that match the generated query.
- Reference Aggregation: A Multi-Head Cross-Attention (MHA) layer processes the current observation alongside the retrieved trajectories. It "attends" to the most pertinent parts of the past to produce a unified reference vector.
This reference vector provides the agent's actor and critic networks with a stationary context of its own past. To ensure practical utility, the authors employ a distillation step, where the complex SR-augmented policy is compressed into a standard network architecture. This allows the performance benefits of SR to be realized during deployment without any additional computational overhead.
Experiments and Results
We evaluated SR across the Unsupervised Reinforcement Learning Benchmark (URLB), covering diverse environments such as Walker, Quadruped, and Jaco-Arm. The module was added to several baseline algorithms across three categories: knowledge-based (e.g., RND, ICM), data-based (e.g., ProtoRL, APT), and competence-based (e.g., APS, CIC).
Key Performance Metrics
SR consistently achieved state-of-the-art results for model-free methods. Notable improvements include:
- Performance Gains: Adding SR to algorithms like APS and ProtoRL resulted in an IQM improvement of up to 17% and a reduction in the Optimality Gap by up to 31%.
- Sample Efficiency: In the complex Quadruped environment, SR-equipped agents reached high performance levels within 100k pretraining steps, a feat vanilla baselines struggled to achieve even with significantly more training.
- Distillation Success: The distillation process transferred skills to standard networks with 96-100% accuracy, confirming that the benefits of self-reference can be deployed with zero runtime cost.
Interestingly, the study also revealed that many baseline URL algorithms perform better with smaller neural networks (latent dimension of 256 vs. 1024), highlighting the sensitivity of these methods to implementation details.
Conclusion and Future Work
Self-Reference provides a powerful mechanism for stabilizing Unsupervised RL by grounding the agent in its own historical context. While highly effective, current limitations include the use of a single query and dependency on raw state-space values for retrieval. Future research will explore multi-modal querying and performing retrieval in learned latent spaces to extend SR's applicability to pixel-based environments and even more complex task distributions.
Summary updated