Meta-learning curiosity algorithms

Ferran Alet, Martin F. Schneider, Tomás Lozano-Pérez, Leslie Pack Kaelbling

Introduction

In reinforcement learning (RL), agents must explore their environments to discover rewards. Because extrinsic rewards are often sparse, researchers hand-design "curiosity" or intrinsic motivation algorithms to provide proxy rewards that encourage exploration. However, hand-designing these strategies is difficult and may not yield optimal exploration behaviors for all environments.

While meta-RL has attempted to learn exploration strategies automatically, it typically does so by transferring neural network weights. This restricts generalization to very similar tasks. This paper tackles the challenge of broad generalization: meta-learning a curiosity mechanism in a simple environment that successfully guides exploration in completely different environments with different state and action spaces.

Meta-Learning Framework

The authors frame curiosity generation as a meta-learning problem with two loops. The inner loop involves a standard RL agent (PPO) learning to maximize a proxy reward signal. The outer loop uses an evolutionary-style search to evaluate different curiosity programs that generate this proxy reward, aiming to maximize the agent's lifetime performance.

The meta-learning framework consists of an inner loop where a fixed RL algorithm (PPO) interacts with an environment, and an outer loop that optimizes the curiosity algorithm program.

Domain-Specific Language for Curiosity

To achieve broad generalization, the authors meta-learn algorithms instead of weights. They designed a Domain-Specific Language (DSL) where programs are Directed Acyclic Graphs (DAGs) of modules, including neural networks, buffers, and loss functions.

Crucially, the DSL uses polymorphic data types. If a program passes a state to a neural network module, the module automatically instantiates as a Convolutional Neural Network (CNN) if the state is an image, or a Multi-Layer Perceptron (MLP) if the state is a vector. This allows the same algorithm to operate on a 2D pixel grid or a 3D physics simulation.

Curiosity algorithms are represented as Directed Acyclic Graphs (DAGs) of modules. The search space includes human-designed baselines like RND and Inverse Features.

To optimize the search, the authors pruned invalid programs and used a $k$-Nearest Neighbor regressor to predict a program's performance based on its syntactic structure, prioritizing the most promising code.

Key Insights and Results

The search yielded two novel, highly effective curiosity algorithms. One, named FAST (Fast Action-Space Transition), is deceptively simple, using only five operations. The search also discovered a complex "Cycle-Consistency" algorithm involving weight-sharing and forward/backward models, demonstrating that automated search can yield non-obvious ideas.

Key results include:

  • Search Efficiency: The predictive $k$-NN model allowed the authors to find 88% of the top 1% of programs while evaluating only half of the search space.
  • Radical Transfer: Programs meta-trained on a simple image-based GridWorld successfully transferred to vector-based environments like Lunar Lander and Acrobot in over 90% of cases.
  • State-of-the-Art Performance: On complex MuJoCo tasks (Ant and Hopper), the meta-learned algorithms significantly outperformed weak baselines and performed on par with top human-designed algorithms like RND and Disagreement.

Conclusion

By searching over programs rather than weights, this work demonstrates a path toward meta-learning general-purpose algorithmic components. While the search space was restricted to small programs, the success of discovered algorithms like FAST suggests that simple, interpretable code can provide powerful exploration priors across diverse domains. Future work may extend this approach to meta-learning optimization algorithms or more complex hierarchical behaviors.

Summary updated

BibTeX
@inproceedings{alet2020meta,
  author = {Ferran Alet and Martin F. Schneider and Tomás Lozano-Pérez and Leslie Pack Kaelbling},
  title = {{Meta-learning curiosity algorithms}},
  booktitle = {International Conference on Learning Representations},
  year = {2020},
  url = {https://openreview.net/forum?id=t9C6Wlm6Nw5},
}
Figure