Introduction
In reinforcement learning (RL), agents must explore their environments to discover rewards. Because extrinsic rewards are often sparse, researchers hand-design "curiosity" or intrinsic motivation algorithms to provide proxy rewards that encourage exploration. However, hand-designing these strategies is difficult and may not yield optimal exploration behaviors for all environments.
While meta-RL has attempted to learn exploration strategies automatically, it typically does so by transferring neural network weights. This restricts generalization to very similar tasks. This paper tackles the challenge of broad generalization: meta-learning a curiosity mechanism in a simple environment that successfully guides exploration in completely different environments with different state and action spaces.
Meta-Learning Framework
The authors frame curiosity generation as a meta-learning problem with two loops. The inner loop involves a standard RL agent (PPO) learning to maximize a proxy reward signal. The outer loop uses an evolutionary-style search to evaluate different curiosity programs that generate this proxy reward, aiming to maximize the agent's lifetime performance.
Domain-Specific Language for Curiosity
To achieve broad generalization, the authors meta-learn algorithms instead of weights. They designed a Domain-Specific Language (DSL) where programs are Directed Acyclic Graphs (DAGs) of modules, including neural networks, buffers, and loss functions.
Crucially, the DSL uses polymorphic data types. If a program passes a state to a neural network module, the module automatically instantiates as a Convolutional Neural Network (CNN) if the state is an image, or a Multi-Layer Perceptron (MLP) if the state is a vector. This allows the same algorithm to operate on a 2D pixel grid or a 3D physics simulation.
To optimize the search, the authors pruned invalid programs and used a $k$-Nearest Neighbor regressor to predict a program's performance based on its syntactic structure, prioritizing the most promising code.
Key Insights and Results
The search yielded two novel, highly effective curiosity algorithms. One, named FAST (Fast Action-Space Transition), is deceptively simple, using only five operations. The search also discovered a complex "Cycle-Consistency" algorithm involving weight-sharing and forward/backward models, demonstrating that automated search can yield non-obvious ideas.
Key results include:
- Search Efficiency: The predictive $k$-NN model allowed the authors to find 88% of the top 1% of programs while evaluating only half of the search space.
- Radical Transfer: Programs meta-trained on a simple image-based GridWorld successfully transferred to vector-based environments like Lunar Lander and Acrobot in over 90% of cases.
- State-of-the-Art Performance: On complex MuJoCo tasks (Ant and Hopper), the meta-learned algorithms significantly outperformed weak baselines and performed on par with top human-designed algorithms like RND and Disagreement.
Conclusion
By searching over programs rather than weights, this work demonstrates a path toward meta-learning general-purpose algorithmic components. While the search space was restricted to small programs, the success of discovered algorithms like FAST suggests that simple, interpretable code can provide powerful exploration priors across diverse domains. Future work may extend this approach to meta-learning optimization algorithms or more complex hierarchical behaviors.
Summary updated