Introduction
Model-Based Reinforcement Learning (MBRL) typically relies on learning a world model via Maximum Likelihood Estimation (MLE). While MLE aims for global accuracy, it often fails in complex environments where model capacity is limited. In such "misspecified" scenarios, the model may waste its representational power on regions irrelevant to the current policy's improvement. Gradient-Aware Model-Based Policy Search (GAMPS) addresses this by prioritizing transitions that are most critical for estimating the policy gradient.
The GAMPS Framework
GAMPS introduces a "decision-aware" approach to policy search. Instead of treating model learning and policy optimization as independent steps, it couples them by weighting the model's training objective based on the needs of the current policy.
1. Gradient-Aware Model Learning
The core innovation is the use of a weighted log-likelihood for training the model $\hat{p}$. The weights $\omega(s, a)$ are derived from a theoretical bound on the policy gradient error. These weights prioritize transitions based on two primary factors:
- Reachability: How likely the current policy is to visit the state-action pair (the state-action distribution).
- Policy Sensitivity: The magnitude of the policy's score function ($\nabla_\theta \log \pi$), indicating regions where the policy has the highest potential for improvement.
2. Model-Value-based Gradient (MVG)
Once the model is learned, GAMPS uses it to compute a value function ($Q$-function) through imaginary rollouts. The policy gradient is then estimated using the MVG estimator, which combines real-world state-action distributions with these model-derived $Q$-values. This approach provides a middle ground between high-variance Monte Carlo methods and high-bias pure model-based methods.
Empirical Evaluation
The authors evaluated GAMPS across three benchmarks designed to test its robustness to model misspecification and its sample efficiency.
- Two-areas Gridworld: A $5 \times 5$ grid where different zones have distinct physics. GAMPS demonstrated the ability to ignore dynamics in zones where the policy was already optimal, focusing its capacity on the "learning frontier."
- Minigolf: A continuous control task involving varying friction (sand vs. grass). GAMPS effectively handled the complex dynamics by prioritizing the transitions necessary for successful putts.
- 3-link Swimmer (MuJoCo): In this high-dimensional continuous task, GAMPS outperformed standard model-free and model-based baselines, particularly in low-data regimes.
Key Insights
- Handling Misspecification: GAMPS excels when the model class is too simple to capture the entire environment. By focusing on "critical" transitions, it achieves better policy updates than a globally "accurate" but locally "blunt" MLE model.
- Sample Efficiency: By making every real-world sample count toward the most relevant parts of the model, GAMPS reaches near-optimal performance significantly faster than traditional methods. In the Gridworld task, it achieved near-optimality with only 10 trajectories.
Conclusion and Future Work
GAMPS provides a robust framework for decision-aware model learning in policy search. While it shows superior performance, future work is needed to reduce the computational overhead of retraining the model at every iteration and to extend the approach to fully online, interactive settings where data collection and model updates happen simultaneously.
Summary updated