Abstract
Hyperparameter tuning remains a significant bottleneck in Reinforcement Learning (RL), where performance is often highly sensitive to choices like the trust region size. This paper proposes a meta-learning framework that treats the policy optimization process as a Markov Decision Process (Meta-MDP). By employing offline RL via Fitted-Q Iteration (FQI), we develop a meta-model capable of dynamically adjusting the trust region hyperparameter $\lambda$ throughout training. Our results demonstrate that this approach not only outperforms fixed-parameter baselines but also generalizes across different policy architectures, effectively teaching RL algorithms to tune themselves.
Introduction
In Policy Gradient methods, the trust region size ($\lambda$) is a critical hyperparameter that balances exploration and stability. Algorithms like Trust Region Policy Optimization (TRPO) rely on $\lambda$ to prevent "catastrophic forgetting"—large, detrimental updates to the agent's policy. Traditionally, $\lambda$ is a static value determined through extensive grid search. However, the optimal step size likely varies as training progresses: larger steps may be beneficial early on for exploration, while smaller steps are necessary for fine-tuning as the policy converges. This work introduces an automated, adaptive method for tuning $\lambda$ using meta-reinforcement learning.
Methodology
The Meta-MDP Framework
The core contribution is modeling the training process itself as a Meta-MDP. In this framework, the Meta-State is a vector of meta-features representing the current state of the learning process. These features include information-theoretic measures and reward statistics. The Meta-Action is the selection of the trust region hyperparameter $\lambda$, and the Meta-Reward is the improvement in the agent's performance following an update.
As shown in Table 1, the meta-features are derived from distributions of states, actions, and rewards at different time steps ($t$ and $t-1$). Measures such as Shannon Entropy ($\mathbb{H}$) and KL-Divergence ($\mathbb{D}_{KL}$) allow the meta-model to gauge the stability and confidence of the current policy.
Learning the Meta-Policy
To solve the Meta-MDP, we utilize Fitted-Q Iteration (FQI), an offline RL algorithm. We first collect a dataset of training trajectories where $\lambda$ is sampled randomly. FQI then learns a Q-function that predicts the long-term performance gain for choosing a specific $\lambda$ given the current meta-features. We also introduce xTRPO, a modified version of TRPO where standard internal safety heuristics are disabled, forcing the algorithm to rely entirely on the $\lambda$ values provided by the meta-model.
Experimental Setup
The framework was evaluated on three distinct environments: Elikoeidis (a single-state environment with a non-linear reward function), Minigolf (a continuous control task), and Cartpole (a standard benchmark). The meta-model was trained on datasets containing 7,000 to 15,000 meta-tuples, capturing the dynamics of the policy optimization process across various initializations.
Results and Discussion
Performance and Adaptivity
The meta-model consistently matched or outperformed the best-tuned fixed hyperparameters. In the Elikoeidis environment, the meta-learned schedule significantly exceeded the performance of the optimal fixed $\lambda$ found via exhaustive grid search. Analysis of the learned meta-policy revealed an emergent "decay" schedule, where the model automatically selects larger trust regions during early training and smaller ones as the policy matures.
Cross-Architecture Generalization
A significant finding is the meta-model's ability to generalize. A model trained using simple linear policies was successfully applied to optimize deep Multi-Layer Perceptrons (MLPs). This suggests that the relationship between behavioral statistics (meta-features) and optimal update steps is somewhat universal, independent of the underlying policy's parameterization.
Conclusion
This work demonstrates that meta-learning can effectively automate the tuning of sensitive RL hyperparameters. By framing optimization as a Meta-MDP, we can learn adaptive schedules that outperform manual tuning and generalize to unseen architectures. Future work will explore scaling these meta-features to high-dimensional state spaces and transitioning from offline to online meta-learning.
Summary updated