Introduction
Reinforcement learning (RL) agents frequently struggle in "sparse reward" environments where the goal is difficult to find through random exploration. To overcome this, researchers often employ curiosity (or intrinsic motivation) to provide agents with proxy rewards for exploring novel states. However, designing these curiosity signals has historically been a manual, trial-and-error process.
While existing "Meta-RL" approaches attempt to automate this by transferring neural network weights, they typically suffer from a lack of generalization, only succeeding when training and testing tasks are nearly identical. This paper tackles the challenge of radical generalization: discovering a curiosity mechanism that works just as well for an agent navigating a maze via pixel inputs as it does for an agent controlling joint torques on a complex robotic ant. Instead of hand-crafting these rewards, the authors propose meta-learning them as programs.
Methodology: Algorithms as Code
The core innovation of this work is representing curiosity mechanisms as programs rather than static neural network weights. By searching for programs, the system can discover fundamental algorithmic patterns (e.g., "predict the next state and reward the error") that are independent of specific task dimensions.
To achieve this, the authors designed a Domain-Specific Language (DSL) consisting of building blocks like neural networks, buffers (FIFO queues), nearest-neighbor modules, and loss functions. These components are combined into Directed Acyclic Graphs (DAGs).
Crucially, these modules use polymorphic abstraction to ensure a program can run in any environment. For instance, a "Neural Network" block automatically instantiates as a Convolutional Neural Network (CNN) if the input is an image, or a Multi-Layer Perceptron (MLP) if the input is a vector.
The Two-Loop Optimization Architecture
The search for optimal curiosity programs is structured as a two-loop optimization process:
- Inner Loop: A standard RL agent (using Proximal Policy Optimization, or PPO) learns a specific task using the intrinsic rewards generated by a candidate curiosity program.
- Outer Loop: An evolutionary-style search evaluates programs based on the agent's total lifetime extrinsic reward, seeking to maximize overall performance.
Because searching through approximately 52,000 possible programs is computationally expensive, the authors employed pruning strategies. They used cheap filtering by testing programs first in a simple 2D GridWorld, and utilized a $k$-nearest neighbor regressor to predict a program's success based on its code structure—essentially performing "meta-meta-learning" to skip thousands of unpromising evaluations.
Key Discoveries and Results
The authors evaluated their approach across a diverse set of environments, including image-based GridWorld navigation, Classic Control (Acrobot, Lunar Lander), and high-dimensional continuous MuJoCo tasks (Ant, Hopper).
- Radical Generalization: Programs discovered in the simple 2D GridWorld transferred successfully to complex 3D robots like the Ant and Hopper without any modification.
- Novel "Aha!" Algorithms: The search discovered two particularly effective novel algorithms. The most notable is FAST (Fast Action-Space Transition), which predicts the action taken between two states and rewards the agent when its prediction diverges significantly from a baseline. Another discovery utilized ideas similar to cycle-consistency in GANs to drive exploration.
- State-of-the-Art Performance: The top meta-learned algorithms performed on par with, or better than, famous human-designed baselines like Random Network Distillation (RND) and the Intrinsic Curiosity Module (ICM).
- Search Efficiency: Thanks to the optimized search strategy, the system found 88% of the top-performing programs while only evaluating half of the total search space.
Limitations and Future Work
While groundbreaking, the approach has limitations. To keep the search space manageable, programs were restricted to a maximum of 7 operations, which may preclude the discovery of even more sophisticated algorithms. Furthermore, the paper primarily focuses on the intrinsic reward itself; how to best dynamically combine this with the environment's actual extrinsic reward remains a promising area for co-adaptation. Finally, despite pruning, the search required 13 GPU days. Scaling this to larger program spaces will necessitate even more advanced search strategies or improved performance predictors.
Summary updated