Trust Region Meta Learning for Policy Optimization

Manuel Occorso, Luca Sabbioni, Alberto Maria Metelli, Marcello RestelliMeta-Knowledge Transfer @ ECML/PKDD 2022 · OpenReview

Abstract

Hyperparameter tuning remains a significant bottleneck in Reinforcement Learning (RL), where performance is often highly sensitive to choices like the trust region size. This paper proposes a meta-learning framework that treats the policy optimization process as a Markov Decision Process (Meta-MDP). By employing offline RL via Fitted-Q Iteration (FQI), we develop a meta-model capable of dynamically adjusting the trust region hyperparameter $\lambda$ throughout training. Our results demonstrate that this approach not only outperforms fixed-parameter baselines but also generalizes across different policy architectures, effectively teaching RL algorithms to tune themselves.

Introduction

In Policy Gradient methods, the trust region size ($\lambda$) is a critical hyperparameter that balances exploration and stability. Algorithms like Trust Region Policy Optimization (TRPO) rely on $\lambda$ to prevent "catastrophic forgetting"—large, detrimental updates to the agent's policy. Traditionally, $\lambda$ is a static value determined through extensive grid search. However, the optimal step size likely varies as training progresses: larger steps may be beneficial early on for exploration, while smaller steps are necessary for fine-tuning as the policy converges. This work introduces an automated, adaptive method for tuning $\lambda$ using meta-reinforcement learning.

Methodology

The Meta-MDP Framework

The core contribution is modeling the training process itself as a Meta-MDP. In this framework, the Meta-State is a vector of meta-features representing the current state of the learning process. These features include information-theoretic measures and reward statistics. The Meta-Action is the selection of the trust region hyperparameter $\lambda$, and the Meta-Reward is the improvement in the agent's performance following an update.

Table 1: Meta-features used to characterize the state of the policy optimization process.

As shown in Table 1, the meta-features are derived from distributions of states, actions, and rewards at different time steps ($t$ and $t-1$). Measures such as Shannon Entropy ($\mathbb{H}$) and KL-Divergence ($\mathbb{D}_{KL}$) allow the meta-model to gauge the stability and confidence of the current policy.

Learning the Meta-Policy

To solve the Meta-MDP, we utilize Fitted-Q Iteration (FQI), an offline RL algorithm. We first collect a dataset of training trajectories where $\lambda$ is sampled randomly. FQI then learns a Q-function that predicts the long-term performance gain for choosing a specific $\lambda$ given the current meta-features. We also introduce xTRPO, a modified version of TRPO where standard internal safety heuristics are disabled, forcing the algorithm to rely entirely on the $\lambda$ values provided by the meta-model.

Experimental Setup

The framework was evaluated on three distinct environments: Elikoeidis (a single-state environment with a non-linear reward function), Minigolf (a continuous control task), and Cartpole (a standard benchmark). The meta-model was trained on datasets containing 7,000 to 15,000 meta-tuples, capturing the dynamics of the policy optimization process across various initializations.

Results and Discussion

Performance and Adaptivity

The meta-model consistently matched or outperformed the best-tuned fixed hyperparameters. In the Elikoeidis environment, the meta-learned schedule significantly exceeded the performance of the optimal fixed $\lambda$ found via exhaustive grid search. Analysis of the learned meta-policy revealed an emergent "decay" schedule, where the model automatically selects larger trust regions during early training and smaller ones as the policy matures.

Cross-Architecture Generalization

A significant finding is the meta-model's ability to generalize. A model trained using simple linear policies was successfully applied to optimize deep Multi-Layer Perceptrons (MLPs). This suggests that the relationship between behavioral statistics (meta-features) and optimal update steps is somewhat universal, independent of the underlying policy's parameterization.

Conclusion

This work demonstrates that meta-learning can effectively automate the tuning of sensitive RL hyperparameters. By framing optimization as a Meta-MDP, we can learn adaptive schedules that outperform manual tuning and generalize to unseen architectures. Future work will explore scaling these meta-features to high-dimensional state spaces and transitioning from offline to online meta-learning.

Summary updated

Abstract

Reinforcement Learning aims to train autonomous agents in their interaction with the environment by means of maximizing a given reward signal; in the last decade there has been an explosion of new algorithms, which make extensive use of hyper-parameters to control their behaviour, accuracy and speed. Often those hyper-parameters are fine-tuned by hand, and the selected values may change drastically the learning performance of the algorithm; furthermore, it happens to train multiple agents on very similar problems, starting from scratch each time. Our goal is to design a Meta-Reinforcement Learning algorithm to optimize the hyper-parameter of a well-known RL algorithm, named Trust Region Policy Optimization. We use knowledge from previous learning sessions and another RL algorithm, Fitted-Q Iteration, to build a policy-agnostic Meta-Model capable to predict the optimal hyper-parameter for TRPO at each of its steps, on new unseen problems, generalizing across different tasks and policy spaces.
BibTeX
@inproceedings{occorso2022trust,
  author = {Manuel Occorso and Luca Sabbioni and Alberto Maria Metelli and Marcello Restelli},
  title = {{Trust Region Meta Learning for Policy Optimization}},
  booktitle = {Meta-Knowledge Transfer @ ECML/PKDD},
  year = {2022},
  url = {https://openreview.net/forum?id=YYTWBrEr4x},
}
Figure