Weighted model estimation for offline model-based reinforcement learning

Toru Hishinuma, Kei SendaNeurIPS 2021

TL;DR

This paper tackles the challenge of learning accurate models for reinforcement learning from fixed datasets (offline RL), especially when the learned policy might behave differently from the data collection policy (covariate shift). They propose a novel "artificial weight" to better estimate the model, which is shown to improve policy evaluation and optimization compared to standard methods.

What Problem Does This Paper Address?

Offline Reinforcement Learning (RL) aims to learn optimal policies from pre-collected datasets without further interaction with the environment. A key approach is Model-Based RL (MBRL), where a model of the environment's dynamics is learned first, and then used to train a policy. However, a major hurdle in offline MBRL is "covariate shift." This occurs when the data distribution collected by the initial policy differs from the data distribution that would be generated by an improved policy. Standard model learning methods (like empirical risk minimization) struggle with this shift, leading to inaccurate models that can't reliably predict the behavior of a new policy. This paper proposes a method to mitigate this issue by weighting the model estimation process.

Methods

The core of this paper is the introduction of an "artificial weight" for model estimation. The authors argue that a "natural weight" would involve the ratio of the true future data distribution to the offline data distribution, but this is impossible to estimate in offline settings. Instead, they propose using the ratio of a simulated future data distribution (generated using the current model and policy) to the offline data distribution. This ratio, the "artificial weight," can be estimated using standard density ratio estimation techniques.

This artificial weight is then incorporated into the model learning objective. Specifically, they define a weighted loss function for model estimation that aims to minimize policy evaluation error. The paper presents algorithms for both policy evaluation (Algorithm 1 and 2) and policy optimization (Algorithm 3), which iteratively update the model and the policy. Algorithm 3 uses an EM-style approach, alternating between optimizing the model (E-step) and the policy (M-step). A key aspect is that the weighting is applied to the model estimation process, which then influences policy optimization.

Datasets

The paper uses the following datasets:

  • Pendulum environment: Offline data is collected using a random policy, and simulated future data is generated using a target controller policy.
  • D4RL Benchmark: Specifically, the Walker2d-medium-expert and Walker2d-medium datasets are used for policy optimization. These are standard benchmarks for offline RL.

Key Insights and Ideas

  • Artificial Weight for Covariate Shift: The central idea is that by weighting the model estimation process with an "artificial weight" derived from simulated future data, we can better account for covariate shift. This is a clever workaround for the impossibility of knowing the true future data distribution.
  • Justification as Upper Bound: The paper provides theoretical justification for using this artificial weight, showing that it relates to evaluating an upper bound of the policy evaluation error. This gives confidence in the approach even though the weight isn't "natural."
  • Model-centric Improvement: The proposed method focuses on improving the model itself to handle covariate shift, which then leads to better policy evaluation and optimization. This is a different approach than some methods that directly try to make the policy robust to model errors.
  • Algorithm 3: EM-style Optimization: The proposed Algorithm 3, which alternates between model and policy optimization, is a structured way to leverage the weighted model estimation for end-to-end policy improvement.

Key Results

  • Pendulum Task: In the pendulum task, Algorithm 1 (full version) significantly improved the accuracy of policy evaluation compared to standard ERM, estimating the discounted return much closer to the true value. Figure 1 visually demonstrates that the weighted model leads to more accurate predictions of the swing-up behavior.
  • D4RL Benchmark (Walker2d-medium-expert): For the Walker2d-medium-expert dataset, the proposed Algorithm 3 with the artificial weight (α=0.2) achieved significantly higher normalized scores than the baseline (α=0, equivalent to MOPO). This indicates better policy optimization. The authors attribute this to the weighted model more accurately predicting the initial moves, preventing falls.
  • D4RL Benchmark (Walker2d-medium): For the Walker2d-medium dataset, the results were mixed, with the weighted approach performing worse in some runs. The authors suggest this might be due to high variance in the real returns for this specific dataset.
  • Simplified Algorithm: The simplified version of the algorithm (Algorithm 2) also showed improvements over ERM in the pendulum task, demonstrating the practical utility of the weighting approach even without theoretical convergence guarantees.

Limitations and Future Work

  • Computational Cost: The full version of Algorithm 1 is computationally expensive, limiting its applicability to large-scale problems. This led to the development of the simplified Algorithm 2.
  • Simplified Algorithm Convergence: Algorithm 2, while practical, lacks theoretical convergence guarantees for the policy optimization objective.
  • Markov Assumption and Reward Design: Like many offline MBRL methods, the approach is subject to limitations related to the Markov assumption and reward function design.
  • Extrapolation: The paper notes that while interpolation and near-extrapolation can be handled by importance weighting, far extrapolation might require additional ideas, such as structured priors.
  • Bayesian MBRL: The authors suggest extending the work to Bayesian MBRL, where a weighted posterior considering covariate shift could be beneficial.

Summary updated

BibTeX
@inproceedings{hishinuma2021weighted,
  author = {Toru Hishinuma and Kei Senda},
  title = {{Weighted model estimation for offline model-based reinforcement learning}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2021},
}
Figure