Multi-Step Generalized Policy Improvement by Leveraging Approximate Models

Lucas Nunes Alegre, Ana L. C. Bazzan, Ann Nowé, Bruno C. da SilvaNeurIPS 2023 · OpenReview

TL;DR

The paper introduces $h$-GPI, a multi-step extension of Generalized Policy Improvement that bridges the gap between model-free transfer and model-based planning. By using an approximate environment model to look ahead $h$ steps and then "bootstrapping" from a library of existing policies, the agent achieves superior zero-shot transfer performance and becomes significantly more robust to errors in its policy library.

What Problem Does This Paper Address?

In Reinforcement Learning (RL), zero-shot transfer is the "holy grail": the ability for an agent to solve a brand-new task immediately, without any further learning, by leveraging prior knowledge.

Currently, two main approaches exist:

  1. Model-Free (GPI + Successor Features): These methods are fast. They look at a library of previously learned policies and pick the best action for the current task. However, they are limited by the "coverage" of the library; if no previous policy is a good fit for the new task, the agent performs poorly.
  2. Model-Based (Planning): These methods use a model of the world to simulate the future. They are flexible but computationally expensive. Crucially, small errors in the model compound over long horizons, leading to "catastrophic" failures in complex tasks.

The authors address the gap between these extremes, seeking a principled way to use approximate models to improve transfer without falling into the trap of compounding model errors.

Methods

The authors propose $h$-GPI (Multi-Step Generalized Policy Improvement). The core idea is an interpolation controlled by a parameter $h$:

  • The Lookahead: The agent uses an approximate model to "unroll" the environment for $h$ steps into the future.
  • The Bootstrapping: At the end of those $h$ steps, instead of planning forever, the agent uses Generalized Policy Improvement (GPI) to estimate the remaining value. It looks at its library of policies and asks, "From this future state, which of my old policies would perform best on this new task?"
  • The Spectrum: When $h=0$, the method is standard model-free GPI. As $h \to \infty$, it approaches full model-based planning.

The paper also introduces a practical implementation using Universal Successor Features Approximators (USFAs) for continuous states and an ensemble of probabilistic neural networks to learn the environment model.

Datasets

The researchers evaluated $h$-GPI on three distinct environments:

  1. Stochastic FourRoom: A tabular gridworld where the agent must collect objects. They added 10% transition noise to make it more challenging.
  2. Reacher: A continuous-state domain involving a two-joint robotic arm that must reach various targets.
  3. FetchPush: A high-dimensional (19D) and complex environment where a robotic arm must push a block to target locations on a table.

Key Insights and Ideas

  • Mitigating Sub-optimality: A major "aha moment" is the proof that as $h$ increases, the agent's performance becomes arbitrarily less susceptible to sub-optimality in its policy library. Essentially, a little bit of planning can compensate for a mediocre set of starting policies.
  • The Error Trade-off: The paper formalizes a fundamental trade-off. Increasing $h$ reduces the impact of value function approximation errors (from the library) but increases the impact of model approximation errors (compounding over the $h$ steps).
  • Strictly Better Bounds: The authors prove that $h$-GPI’s performance lower bound is strictly better than standard GPI’s (assuming a perfect model). They provide new, generalized bounds that account for errors in both the model and the value functions simultaneously.

Key Results

  • Superior Performance: In all domains, $h$-GPI consistently outperformed standard GPI and state-of-the-art baselines like Constrained GPI (CGPI).
  • Efficiency: In the FourRoom domain, $h$-GPI with a tiny lookahead ($h=2$) matched the performance of a pure planning agent (SF-MPC) that was allowed to look ahead ten times further ($h=20$).
Performance comparison in the FourRoom domain showing $h$-GPI outperforming standard GPI and matching longer-horizon MPC.
  • Robustness to Noise: When the researchers artificially added Gaussian noise to the policy library's value functions, standard GPI's performance collapsed. $h$-GPI, however, maintained high performance by relying more on its model-based lookahead to "see past" the noisy estimates.
  • Scaling: The method successfully scaled to the high-dimensional FetchPush task, showing that it can handle complex dynamics and continuous state spaces.

Limitations and Future Work

  • Computational Overhead: Planning for $h$ steps is more expensive than a single-step lookup. For very large $h$ or complex models, the real-time decision-making speed could decrease.
  • Fixed Horizon: The current work uses a fixed $h$. The authors suggest that a "principled method to dynamically select the best value of $h$" based on current model uncertainty would be a valuable next step.
  • Model Dependency: While $h$-GPI is more robust than pure planning, it still requires a reasonably good model. If the model is extremely poor, the $h$-step lookahead could lead the agent astray. Integrating uncertainty estimation (e.g., implicit value ensembles) into the $h$-GPI framework is proposed as future work to further mitigate this.

Summary updated

Abstract

We introduce a principled method for performing zero-shot transfer in reinforcement learning (RL) by exploiting approximate models of the environment. Zero-shot transfer in RL has been investigated by leveraging methods rooted in generalized policy improvement (GPI) and successor features (SFs). Although computationally efficient, these methods are model-free: they analyze a library of policies---each solving a particular task---and identify which action the agent should take. We investigate the more general setting where, in addition to a library of policies, the agent has access to an approximate environment model. Even though model-based RL algorithms can identify near-optimal policies, they are typically computationally intensive. We introduce $h$-GPI, a multi-step extension of GPI that interpolates between these extremes---standard model-free GPI and fully model-based planning---as a function of a parameter, $h$, regulating the amount of time the agent has to reason. We prove that $h$-GPI's performance lower bound is strictly better than GPI's, and show that $h$-GPI generally outperforms GPI as $h$ increases. Furthermore, we prove that as $h$ increases, $h$-GPI's performance becomes arbitrarily less susceptible to sub-optimality in the agent's policy library. Finally, we introduce novel bounds characterizing the gains achievable by $h$-GPI as a function of approximation errors in both the agent's policy library and its (possibly learned) model. These bounds strictly generalize those known in the literature. We evaluate $h$-GPI on challenging tabular and continuous-state problems under value function approximation and show that it consistently outperforms GPI and state-of-the-art competing methods under various levels of approximation errors.
BibTeX
@inproceedings{alegre2023multi,
  author = {Lucas Nunes Alegre and Ana L. C. Bazzan and Ann Nowé and Bruno C. da Silva},
  title = {{Multi-Step Generalized Policy Improvement by Leveraging Approximate Models}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2023},
  url = {https://openreview.net/forum?id=KFj0Q1EXvU},
}
Figure