Meta-learning curiosity algorithms

Ferran Alet, Martin F. Schneider, Tomás Lozano-Pérez, Leslie Pack Kaelbling

Introduction

Reinforcement learning (RL) agents frequently struggle in "sparse reward" environments where the goal is difficult to find through random exploration. To overcome this, researchers often employ curiosity (or intrinsic motivation) to provide agents with proxy rewards for exploring novel states. However, designing these curiosity signals has historically been a manual, trial-and-error process.

While existing "Meta-RL" approaches attempt to automate this by transferring neural network weights, they typically suffer from a lack of generalization, only succeeding when training and testing tasks are nearly identical. This paper tackles the challenge of radical generalization: discovering a curiosity mechanism that works just as well for an agent navigating a maze via pixel inputs as it does for an agent controlling joint torques on a complex robotic ant. Instead of hand-crafting these rewards, the authors propose meta-learning them as programs.

Methodology: Algorithms as Code

The core innovation of this work is representing curiosity mechanisms as programs rather than static neural network weights. By searching for programs, the system can discover fundamental algorithmic patterns (e.g., "predict the next state and reward the error") that are independent of specific task dimensions.

To achieve this, the authors designed a Domain-Specific Language (DSL) consisting of building blocks like neural networks, buffers (FIFO queues), nearest-neighbor modules, and loss functions. These components are combined into Directed Acyclic Graphs (DAGs).

Example of a curiosity algorithm (Random Network Distillation) represented as a program DAG in the authors' DSL.

Crucially, these modules use polymorphic abstraction to ensure a program can run in any environment. For instance, a "Neural Network" block automatically instantiates as a Convolutional Neural Network (CNN) if the input is an image, or a Multi-Layer Perceptron (MLP) if the input is a vector.

The Two-Loop Optimization Architecture

The search for optimal curiosity programs is structured as a two-loop optimization process:

The two-loop meta-learning architecture. The outer loop searches for curiosity programs, while the inner loop uses the generated intrinsic reward to train a standard RL agent.
  • Inner Loop: A standard RL agent (using Proximal Policy Optimization, or PPO) learns a specific task using the intrinsic rewards generated by a candidate curiosity program.
  • Outer Loop: An evolutionary-style search evaluates programs based on the agent's total lifetime extrinsic reward, seeking to maximize overall performance.

Because searching through approximately 52,000 possible programs is computationally expensive, the authors employed pruning strategies. They used cheap filtering by testing programs first in a simple 2D GridWorld, and utilized a $k$-nearest neighbor regressor to predict a program's success based on its code structure—essentially performing "meta-meta-learning" to skip thousands of unpromising evaluations.

Key Discoveries and Results

The authors evaluated their approach across a diverse set of environments, including image-based GridWorld navigation, Classic Control (Acrobot, Lunar Lander), and high-dimensional continuous MuJoCo tasks (Ant, Hopper).

  • Radical Generalization: Programs discovered in the simple 2D GridWorld transferred successfully to complex 3D robots like the Ant and Hopper without any modification.
  • Novel "Aha!" Algorithms: The search discovered two particularly effective novel algorithms. The most notable is FAST (Fast Action-Space Transition), which predicts the action taken between two states and rewards the agent when its prediction diverges significantly from a baseline. Another discovery utilized ideas similar to cycle-consistency in GANs to drive exploration.
  • State-of-the-Art Performance: The top meta-learned algorithms performed on par with, or better than, famous human-designed baselines like Random Network Distillation (RND) and the Intrinsic Curiosity Module (ICM).
  • Search Efficiency: Thanks to the optimized search strategy, the system found 88% of the top-performing programs while only evaluating half of the total search space.

Limitations and Future Work

While groundbreaking, the approach has limitations. To keep the search space manageable, programs were restricted to a maximum of 7 operations, which may preclude the discovery of even more sophisticated algorithms. Furthermore, the paper primarily focuses on the intrinsic reward itself; how to best dynamically combine this with the environment's actual extrinsic reward remains a promising area for co-adaptation. Finally, despite pruning, the search required 13 GPU days. Scaling this to larger program spaces will necessitate even more advanced search strategies or improved performance predictors.

Summary updated

Abstract

We hypothesize that curiosity is a mechanism found by evolution that encourages meaningful exploration early in an agent's life in order to expose it to experiences that enable it to obtain high rewards over the course of its lifetime. We formulate the problem of generating curious behavior as one of meta-learning: an outer loop will search over a space of curiosity mechanisms that dynamically adapt the agent's reward signal, and an inner loop will perform standard reinforcement learning using the adapted reward signal. However, current meta-RL methods based on transferring neural network weights have only generalized between very similar tasks. To broaden the generalization, we instead propose to meta-learn algorithms: pieces of code similar to those designed by humans in ML papers. Our rich language of programs combines neural networks with other building blocks such as buffers, nearest-neighbor modules and custom loss functions. We demonstrate the effectiveness of the approach empirically, finding two novel curiosity algorithms that perform on par or better than human-designed published curiosity algorithms in domains as disparate as grid navigation with image inputs, acrobot, lunar lander, ant and hopper.
BibTeX
@inproceedings{alet2020meta,
  author = {Ferran Alet and Martin F. Schneider and Tomás Lozano-Pérez and Leslie Pack Kaelbling},
  title = {{Meta-learning curiosity algorithms}},
  booktitle = {International Conference on Learning Representations},
  year = {2020},
  url = {https://openreview.net/forum?id=t9C6Wlm6Nw5},
}
Figure