Gradient-Aware Model-based Policy Search.

Pierluca D'Oro, Alberto Maria Metelli, Andrea Tirinzoni, Matteo Papini, Marcello Restelli

Abstract

Traditional model-based reinforcement learning approaches learn a model of the environment dynamics without explicitly considering how it will be used by the agent. In the presence of misspecified model classes, this can lead to poor estimates, as some relevant available information is ignored. In this paper, we introduce a novel model-based policy search approach that exploits the knowledge of the current agent policy to learn an approximate transition model, focusing on the portions of the environment that are most relevant for policy improvement. We leverage a weighting scheme, derived from the minimization of the error on the model-based policy gradient estimator, in order to define a suitable objective function that is optimized for learning the approximate transition model. Then, we integrate this procedure into a batch policy improvement algorithm, named Gradient-Aware Model-based Policy Search (GAMPS), which iteratively learns a transition model and uses it, together with the collected trajectories, to compute the new policy parameters. Finally, we empirically validate GAMPS on benchmark domains analyzing and discussing its properties.

Introduction

In Model-Based Reinforcement Learning (MBRL), the standard paradigm involves learning a transition model of the environment and subsequently using that model to simulate trajectories for policy optimization. Typically, these models are trained using Maximum Likelihood Estimation (MLE), which assigns equal importance to every transition in the training dataset.

However, this approach can be highly inefficient. In complex, real-world environments like robotics, it is often impossible to model the entire world perfectly. When the chosen model class is "misspecified"—meaning it lacks the capacity to capture the full complexity of the environment—MLE tends to average out the dynamics. This results in a model that is mediocre across the entire state space but lacks the precision required in critical regions for effective policy improvement. GAMPS addresses this by asking: if we have a limited modeling "budget," where should we spend it? The proposed solution is to focus on the parts of the environment where the policy gradient is most sensitive to changes.

Methods

The core innovation of Gradient-Aware Model-based Policy Search (GAMPS) is a novel weighting scheme for the model-learning objective. Instead of standard MLE, GAMPS minimizes a weighted log-likelihood where the weights for each transition are derived from a theoretical bound on the policy gradient error.

The Weighting Scheme

The weights consist of three primary components:

  1. Discount Factor ($\gamma^t$): Prioritizes earlier transitions in a trajectory, as they have a more significant impact on the total return.
  2. Importance Weights: Corrects for the distribution mismatch between the data-collection policy and the current policy being optimized.
  3. Score Magnitude: This is the "gradient-aware" component. It assigns higher weights to transitions occurring in regions where the policy has a high "opportunity to be improved"—specifically, where the gradient of the log-policy is large.
Comparison of standard state-action distribution weights (b) versus the proposed gradient-aware weights (c) in a Gridworld task.

As shown in Figure 2, in a Gridworld with heterogeneous dynamics, standard distribution-based weights (b) spread the modeling effort across all visited states. In contrast, the gradient-aware weights (c) concentrate the model's capacity on the specific transitions that are most influential for reaching the goal 'G'.

The GAMPS Algorithm

GAMPS operates iteratively: it collects data from the environment, trains the weighted model, performs "imaginary" rollouts within this model to estimate the Q-function, and then updates the policy using a hybrid Model-Value-based Gradient (MVG) estimator. This estimator uses real-world trajectories to sample states but relies on the learned model to estimate the value of those states.

Experimental Evaluation

The authors evaluated GAMPS across three distinct environments designed to test its robustness to model misspecification and sample efficiency:

  • Two-areas Gridworld: A 5x5 grid with reversed dynamics in different zones, designed to show how MLE fails by averaging conflicting dynamics while GAMPS focuses on the goal-relevant zone.
  • Minigolf: A simulation requiring an agent to hit a ball into a hole across terrains with varying friction levels.
  • 3-link Swimmer: A continuous control benchmark (MuJoCo) where a multi-link robot must learn to swim forward.

Key Results

GAMPS demonstrated significant improvements in both gradient estimation accuracy and overall sample efficiency. In the Gridworld task, GAMPS achieved a cosine similarity of 1.0 with the true policy gradient, indicating a perfect estimation of the improvement direction. In contrast, standard MLE achieved only 0.449, often leading the agent toward suboptimal policies due to poor local modeling.

Average return over iterations for 50 trajectories, comparing GAMPS against various baselines.

In more complex tasks like the Swimmer (Figure 6), GAMPS consistently outperformed both model-free baselines (REINFORCE, PGT) and standard model-based policy search. It reached higher average returns with fewer trajectories, proving that even simple linear models can be highly effective when forced to focus on the most relevant parts of the state space.

Conclusion

GAMPS shifts the focus of model-based RL from global accuracy to local utility. By using gradient sensitivity as a compass, the algorithm identifies bottleneck transitions that most influence policy updates. While the current batch setting requires re-weighting and re-training at every iteration—introducing some computational overhead—the results suggest that gradient awareness is a powerful tool for overcoming model misspecification and improving sample efficiency in reinforcement learning.

Summary updated

Abstract

Traditional model-based reinforcement learning approaches learn a model of the environment dynamics without explicitly considering how it will be used by the agent. In the presence of misspecified model classes, this can lead to poor estimates, as some relevant available information is ignored. In this paper, we introduce a novel model-based policy search approach that exploits the knowledge of the current agent policy to learn an approximate transition model, focusing on the portions of the environment that are most relevant for policy improvement. We leverage a weighting scheme, derived from the minimization of the error on the model-based policy gradient estimator, in order to define a suitable objective function that is optimized for learning the approximate transition model. Then, we integrate this procedure into a batch policy improvement algorithm, named Gradient-Aware Model-based Policy Search (GAMPS), which iteratively learns a transition model and uses it, together with the collected trajectories, to compute the new policy parameters. Finally, we empirically validate GAMPS on benchmark domains analyzing and discussing its properties.
BibTeX
@inproceedings{doro2020gradient,
  author = {Pierluca D'Oro and Alberto Maria Metelli and Andrea Tirinzoni and Matteo Papini and Marcello Restelli},
  title = {{Gradient-Aware Model-based Policy Search.}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2020},
  doi = {10.1609/aaai.v34i04.5791},
}
Figure