Abstract
We reinterpret the problem of finding intrinsic rewards in reinforcement learning (RL) as a bi-level optimization problem. By treating the reward function as a hyperparameter, we leverage recent advancements in hyperparameter optimization to learn effective reward signals dynamically. Our approach introduces a novel Conditional Layer Normalization (CLN) mechanism within Self-Tuning Networks (STNs), enabling the agent to learn "better-than-human" reward signals. We demonstrate that this method achieves superior performance compared to state-of-the-art baselines on complex MuJoCo continuous control tasks.
Introduction
In reinforcement learning, agents are typically trained using extrinsic rewards designed by humans. While intuitive, these signals are often suboptimal for machine learning algorithms, potentially being sparse, noisy, or providing poor gradients. "Intrinsic rewards" serve as auxiliary signals to accelerate learning. However, systematically discovering the right intrinsic reward is challenging. Previous approaches relied on heuristics or computationally expensive meta-gradients. This work addresses the challenge by framing the search for intrinsic rewards as a systematic bi-level optimization problem.
Methodology
Bi-Level Optimization Framework
The core of our approach is a bi-level optimization structure consisting of an inner and outer loop:
- Inner Loop: The policy ($\pi$) is trained to maximize the learned intrinsic reward ($r^\eta$).
- Outer Loop: The parameters of the intrinsic reward function ($\eta$) are updated to maximize the extrinsic reward achieved by the resulting policy.
To avoid the computational burden of differentiating through long update chains, we utilize Self-Tuning Networks (STNs). STNs aim to learn a "best-response function"—a direct mapping from reward parameters to optimal policy weights.
Conditional Layer Normalization (CLN)
A key technical contribution is the Conditional Layer Normalization (CLN) mechanism. Unlike standard Layer Normalization with static gain and bias, CLN dynamically generates these parameters via a linear transformation of the intrinsic reward parameters. This allows the learned reward to directly modulate the policy's behavior at the normalization layer, providing a more stable and potent gradient signal for the meta-learner.
Experimental Results
Benchmarks
We evaluated our method on standard continuous control benchmarks from OpenAI Gym, utilizing the MuJoCo physics simulator: Hopper-v2, Walker2d-v2, Ant-v2, and Humanoid-v2. We focused on the standard "dense" reward versions to demonstrate that learned intrinsic rewards can enhance performance even in well-defined RL environments.
Performance Comparison
Across all tested environments, the CLN-based method consistently outperformed both the baseline PPO algorithm and LIRPG, a previous state-of-the-art reward-learning algorithm.
Notably, the method showed significant gains in the Humanoid-v2 task, which is characterized by high-dimensional state and action spaces. While CLN achieved higher final performance, it occasionally exhibited slower initial convergence, likely due to the overhead of learning the initial mapping between intrinsic rewards and policy weights.
Discussion and Conclusion
Our analysis indicates that learned intrinsic rewards are often more "generous" than extrinsic ones during early training, promoting exploration and preventing premature convergence to local optima. We also found that a simplified "response gradient" yielded better performance than more complex second-order methods, making the algorithm computationally efficient.
Future work will explore combining this bi-level approach with curiosity-based exploratory rewards and extending the framework to truly sparse reward settings, where the meta-learner currently faces challenges due to the lack of initial extrinsic signals.
Summary updated