Gradient-Aware Model-Based Policy Search

Pierluca D'Oro, Alberto Maria Metelli, Andrea Tirinzoni, Matteo Papini, Marcello Restelli

Introduction

Model-Based Reinforcement Learning (MBRL) typically relies on learning a world model via Maximum Likelihood Estimation (MLE). While MLE aims for global accuracy, it often fails in complex environments where model capacity is limited. In such "misspecified" scenarios, the model may waste its representational power on regions irrelevant to the current policy's improvement. Gradient-Aware Model-Based Policy Search (GAMPS) addresses this by prioritizing transitions that are most critical for estimating the policy gradient.

The GAMPS Framework

GAMPS introduces a "decision-aware" approach to policy search. Instead of treating model learning and policy optimization as independent steps, it couples them by weighting the model's training objective based on the needs of the current policy.

1. Gradient-Aware Model Learning

The core innovation is the use of a weighted log-likelihood for training the model $\hat{p}$. The weights $\omega(s, a)$ are derived from a theoretical bound on the policy gradient error. These weights prioritize transitions based on two primary factors:

  • Reachability: How likely the current policy is to visit the state-action pair (the state-action distribution).
  • Policy Sensitivity: The magnitude of the policy's score function ($\nabla_\theta \log \pi$), indicating regions where the policy has the highest potential for improvement.
Empirical weights based on the policy score function, highlighting regions where the policy gradient is most sensitive to model accuracy.

2. Model-Value-based Gradient (MVG)

Once the model is learned, GAMPS uses it to compute a value function ($Q$-function) through imaginary rollouts. The policy gradient is then estimated using the MVG estimator, which combines real-world state-action distributions with these model-derived $Q$-values. This approach provides a middle ground between high-variance Monte Carlo methods and high-bias pure model-based methods.

Empirical Evaluation

The authors evaluated GAMPS across three benchmarks designed to test its robustness to model misspecification and its sample efficiency.

  • Two-areas Gridworld: A $5 \times 5$ grid where different zones have distinct physics. GAMPS demonstrated the ability to ignore dynamics in zones where the policy was already optimal, focusing its capacity on the "learning frontier."
  • Minigolf: A continuous control task involving varying friction (sand vs. grass). GAMPS effectively handled the complex dynamics by prioritizing the transitions necessary for successful putts.
  • 3-link Swimmer (MuJoCo): In this high-dimensional continuous task, GAMPS outperformed standard model-free and model-based baselines, particularly in low-data regimes.
The Gridworld environment (a) and the empirical state-action distribution weights (b) used to focus model learning on transitions reachable from the initial state distribution $\mu$.

Key Insights

  • Handling Misspecification: GAMPS excels when the model class is too simple to capture the entire environment. By focusing on "critical" transitions, it achieves better policy updates than a globally "accurate" but locally "blunt" MLE model.
  • Sample Efficiency: By making every real-world sample count toward the most relevant parts of the model, GAMPS reaches near-optimal performance significantly faster than traditional methods. In the Gridworld task, it achieved near-optimality with only 10 trajectories.

Conclusion and Future Work

GAMPS provides a robust framework for decision-aware model learning in policy search. While it shows superior performance, future work is needed to reduce the computational overhead of retraining the model at every iteration and to extend the approach to fully online, interactive settings where data collection and model updates happen simultaneously.

Summary updated

Abstract

Traditional model-based reinforcement learning approaches learn a model of the environment dynamics without explicitly considering how it will be used by the agent. In the presence of misspecified model classes, this can lead to poor estimates, as some relevant available information is ignored. In this paper, we introduce a novel model-based policy search approach that exploits the knowledge of the current agent policy to learn an approximate transition model, focusing on the portions of the environment that are most relevant for policy improvement. We leverage a weighting scheme, derived from the minimization of the error on the model-based policy gradient estimator, in order to define a suitable objective function that is optimized for learning the approximate transition model. Then, we integrate this procedure into a batch policy improvement algorithm, named Gradient-Aware Model-based Policy Search (GAMPS), which iteratively learns a transition model and uses it, together with the collected trajectories, to compute the new policy parameters. Finally, we empirically validate GAMPS on benchmark domains analyzing and discussing its properties.
BibTeX
@inproceedings{doro2020gradient,
  author = {Pierluca D'Oro and Alberto Maria Metelli and Andrea Tirinzoni and Matteo Papini and Marcello Restelli},
  title = {{Gradient-Aware Model-Based Policy Search}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2020},
  doi = {10.1609/aaai.v34i04.5791},
}
Figure