Augmenting Unsupervised Reinforcement Learning with Self-Reference

Andrew Zhao, Erle Zhu, Rui Lu, Matthieu Lin, Yong-Jin Liu, Gao HuangarXiv 2023 · DOI · arXiv

Abstract

Unsupervised Reinforcement Learning (URL) typically follows a two-stage paradigm of task-agnostic pretraining followed by task-specific finetuning. However, this approach is often hampered by non-stationarity in intrinsic rewards and catastrophic forgetting of exploratory behaviors during finetuning. We introduce Self-Reference (SR), an architectural augmentation that allows agents to explicitly retrieve and aggregate their own historical experiences. By providing temporal context, SR stabilizes the learning process and preserves robust exploration strategies. Our evaluations on the Unsupervised RL Benchmark (URLB) demonstrate that SR achieves state-of-the-art performance, improving the Interquartile Mean (IQM) by up to 17% while allowing for zero-overhead deployment via policy distillation.

Introduction

In Unsupervised Reinforcement Learning, agents are initially tasked with exploring an environment without external rewards, often driven by intrinsic motivations like novelty or surprise. While effective for discovery, this leads to a significant challenge: non-stationarity. As an agent explores, previously "novel" states become familiar, causing the intrinsic reward function to shift constantly. Most current algorithms fail to account for this history, leading to unstable training. Furthermore, during the transition to a downstream task (finetuning), agents often suffer from catastrophic forgetting, where they abandon the diverse behaviors learned during pretraining in favor of narrow, sub-optimal strategies for the specific task.

Methodology: The Self-Reference Module

The Self-Reference (SR) module addresses these challenges by enabling the agent to "look back" at its history. The architecture consists of three core components integrated into the standard RL loop:

  1. Query Module: A neural network trained via Proximal Policy Optimization (PPO) that generates a query based on the current observation. This module learns to dynamically seek historical information—such as previous visits to similar states or potential future trajectories—that is most relevant to the current context.
  2. Retrieval Mechanism: Utilizing the Faiss library for efficient similarity search, the agent retrieves the top k experience sequences (short trajectories) from its replay buffer that match the generated query.
  3. Reference Aggregation: A Multi-Head Cross-Attention (MHA) layer processes the current observation alongside the retrieved trajectories. It "attends" to the most pertinent parts of the past to produce a unified reference vector.
Architecture of the Self-Reference (SR) module and its integration into the Unsupervised RL framework.

This reference vector provides the agent's actor and critic networks with a stationary context of its own past. To ensure practical utility, the authors employ a distillation step, where the complex SR-augmented policy is compressed into a standard network architecture. This allows the performance benefits of SR to be realized during deployment without any additional computational overhead.

Experiments and Results

We evaluated SR across the Unsupervised Reinforcement Learning Benchmark (URLB), covering diverse environments such as Walker, Quadruped, and Jaco-Arm. The module was added to several baseline algorithms across three categories: knowledge-based (e.g., RND, ICM), data-based (e.g., ProtoRL, APT), and competence-based (e.g., APS, CIC).

Key Performance Metrics

SR consistently achieved state-of-the-art results for model-free methods. Notable improvements include:

  • Performance Gains: Adding SR to algorithms like APS and ProtoRL resulted in an IQM improvement of up to 17% and a reduction in the Optimality Gap by up to 31%.
  • Sample Efficiency: In the complex Quadruped environment, SR-equipped agents reached high performance levels within 100k pretraining steps, a feat vanilla baselines struggled to achieve even with significantly more training.
  • Distillation Success: The distillation process transferred skills to standard networks with 96-100% accuracy, confirming that the benefits of self-reference can be deployed with zero runtime cost.
Comprehensive performance comparison showing the impact of SR across various URL algorithm categories and metrics.

Interestingly, the study also revealed that many baseline URL algorithms perform better with smaller neural networks (latent dimension of 256 vs. 1024), highlighting the sensitivity of these methods to implementation details.

Conclusion and Future Work

Self-Reference provides a powerful mechanism for stabilizing Unsupervised RL by grounding the agent in its own historical context. While highly effective, current limitations include the use of a single query and dependency on raw state-space values for retrieval. Future research will explore multi-modal querying and performing retrieval in learned latent spaces to extend SR's applicability to pixel-based environments and even more complex task distributions.

Summary updated

Abstract

Humans possess the ability to draw on past experiences explicitly when learning new tasks and applying them accordingly. We believe this capacity for self-referencing is especially advantageous for reinforcement learning agents in the unsupervised pretrain-then-finetune setting. During pretraining, an agent's past experiences can be explicitly utilized to mitigate the nonstationarity of intrinsic rewards. In the finetuning phase, referencing historical trajectories prevents the unlearning of valuable exploratory behaviors. Motivated by these benefits, we propose the Self-Reference (SR) approach, an add-on module explicitly designed to leverage historical information and enhance agent performance within the pretrain-finetune paradigm. Our approach achieves state-of-the-art results in terms of Interquartile Mean (IQM) performance and Optimality Gap reduction on the Unsupervised Reinforcement Learning Benchmark for model-free methods, recording an 86% IQM and a 16% Optimality Gap. Additionally, it improves current algorithms by up to 17% IQM and reduces the Optimality Gap by 31%. Beyond performance enhancement, the Self-Reference add-on also increases sample efficiency, a crucial attribute for real-world applications.
BibTeX
@misc{zhao2023augmenting,
  author = {Andrew Zhao and Erle Zhu and Rui Lu and Matthieu Lin and Yong-Jin Liu and Gao Huang},
  title = {{Augmenting Unsupervised Reinforcement Learning with Self-Reference}},
  year = {2023},
  eprint = {2311.09692},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2311.09692},
}
Figure