Reincarnating Reinforcement Learning

Rishabh AgarwalOpenReview 2022 · arXiv · OpenReview

TL;DR

This paper introduces "reincarnating RL" (RRL), a new research paradigm that reuses prior computational work (like learned policies) to accelerate RL development, moving away from the inefficient "tabula rasa" (from scratch) approach. They propose a novel algorithm, QDagger, that significantly speeds up training by efficiently transferring suboptimal policies to new agents, demonstrating its effectiveness across diverse tasks.

What Problem Does This Paper Address?

The prevailing approach in Reinforcement Learning (RL) research is to train agents "tabula rasa," meaning from scratch, without leveraging any prior knowledge. This is highly inefficient, especially for large-scale, complex problems where retraining can take weeks or months. This computational barrier also limits access for researchers without industrial-scale resources. Real-world RL systems often undergo frequent changes, and current methods for incorporating these changes are ad hoc and not generalizable. The paper argues for a shift towards a "reincarnating RL" workflow where prior computational effort is reused, making RL research more accessible, efficient, and democratized.

Methods

The core contribution is the concept of reincarnating RL (RRL), which advocates for reusing learned policies and computational work across development iterations or between agents. As a concrete step towards this, the paper focuses on Policy-to-Value Reincarnating RL (PVRL), specifically transferring a suboptimal teacher policy to a value-based student agent.

The authors propose QDagger, a novel algorithm for PVRL. QDagger combines elements of Dagger (an interactive imitation learning algorithm) with n-step Q-learning. It involves an initial pre-training phase using a distillation loss (to mimic the teacher) and a TD loss, followed by an online training phase where the teacher "corrects" the student's mistakes. Crucially, QDagger includes a mechanism to "wean" the student off the teacher policy over time, allowing it to deviate and potentially surpass the teacher.

Datasets

The experiments were conducted on several benchmarks:

  • Atari 2600 games: A subset of 10 commonly-used games (Asterix, Breakout, Space Invaders, Seaquest, Q*Bert, Beam Rider, Enduro, Ms Pacman, Bowling, and River Raid) were used for evaluating PVRL algorithms.
  • Humanoid-run locomotion task: A challenging continuous control task from DeepMind Control Suite.
  • Balloon Learning Environment (BLE): A simulated real-world environment for navigating stratospheric balloons.

For training and evaluation, the authors utilized:

  • Teacher policies: Obtained from pre-trained DQN agents (trained for 400M frames) and other state-of-the-art RL agents.
  • Replay buffers: The final replay buffers from trained teacher agents were used as a source of prior data.

Key Insights and Ideas

  • "Tabula Rasa" is an anomaly, not the norm: The paper convincingly argues that in practice, large-scale RL systems rarely start from scratch. The "tabula rasa" approach is an academic ideal that is often impractical.
  • Democratizing RL: RRL has the potential to significantly lower the barrier to entry for RL research, allowing more researchers to tackle complex problems without massive computational resources.
  • Beyond imitation: PVRL, and specifically QDagger, goes beyond simple imitation learning by aiming to transfer knowledge to a value-based agent and allowing it to eventually surpass the teacher.
  • Weaning is crucial: The ability for a student agent to "wean" itself off a suboptimal teacher is essential for RRL to be truly effective and avoid being perpetually constrained by the teacher's limitations.
  • Teacher quality matters, but so does behavior: The performance of the reincarnated student depends not only on the teacher's absolute performance but also on its specific behavior and how it was trained.

Key Results

  • QDagger outperforms baselines: In the PVRL setting on Atari 2600 games, QDagger significantly outperformed existing methods like JSRL, Rehearsal, Offline RL Pretraining, Kickstarting, and DQfD.
  • Surpassing the teacher: QDagger was able to surpass the performance of the suboptimal teacher policy within 10 million frames on Atari games, a feat not achieved by other methods.
  • Significant speedups: Reincarnating RL via QDagger demonstrated substantial sample and compute efficiency gains. For example, fine-tuning a DQN agent with Adam and 3-step returns achieved the performance of a tabula rasa DQN trained for 400M frames using only 20M frames.
  • Effective across domains: The benefits of RRL were shown on Atari games, a challenging locomotion task (humanoid-run), and a real-world problem (stratospheric balloons).
  • Fine-tuning limitations: While fine-tuning can be effective, it often requires the same network architecture and can lead to performance degradation with prolonged training, highlighting the need for more flexible reincarnation methods.

Limitations and Future Work

  • Teacher dependence: While QDagger aims to wean off the teacher, the initial performance of the reincarnated agent is still dependent on the teacher's quality. The paper notes that a better teacher generally leads to a better student.
  • Generalizability of PVRL: The paper focuses on PVRL, but future work could explore transferring knowledge from policies to other agent types or from other forms of prior computation (e.g., representations, skills).
  • Multiple suboptimal teachers: Extending PVRL to learn from multiple suboptimal teachers is an interesting avenue.
  • Beyond policies: Future research could investigate how to incorporate knowledge from pretrained models, representations, or skills, not just policies.
  • Embodied agents: The authors believe RRL will be crucial for building embodied agents in open-ended domains.
  • Reproducibility: The paper acknowledges that reproducing RRL research can be challenging due to the reliance on prior computational work, advocating for the release of model checkpoints and data.

Summary updated

Abstract

Learning tabula rasa, that is without any prior knowledge, is the prevalent workflow in reinforcement learning (RL) research. However, RL systems, when applied to large-scale settings, rarely operate tabula rasa. Such large-scale systems undergo multiple design or algorithmic changes during their development cycle and use ad hoc approaches for incorporating these changes without re-training from scratch, which would have been prohibitively expensive. Additionally, the inefficiency of deep RL typically excludes researchers without access to industrial-scale resources from tackling computationally-demanding problems. To address these issues, we present reincarnating RL as an alternative workflow, where prior computational work (e.g., learned policies) is reused or transferred between design iterations of an RL agent, or from one RL agent to another. As a step towards enabling reincarnating RL from any agent to any other agent, we focus on the specific setting of efficiently transferring an existing sub-optimal policy to a standalone value-based RL agent. We find that existing approaches fail in this setting and propose a simple algorithm to address their limitations. Equipped with this algorithm, we demonstrate reincarnating RL's gains over tabula rasa RL on Atari 2600 games, a challenging locomotion task, and the real-world problem of navigating stratospheric balloons. Overall, this work argues for an alternative approach to RL research, which we believe could significantly improve real-world RL adoption and help democratize it further.
BibTeX
@misc{agarwal2022reincarnating,
  author = {Rishabh Agarwal},
  title = {{Reincarnating Reinforcement Learning}},
  year = {2022},
  eprint = {2206.01626},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2206.01626},
  howpublished = {OpenReview},
}
Figure