Avoiding Overfitting to the Importance Weights in Offline Policy Optimization

Yao Liu, Emma BrunskillICLR 2022 · OpenReview

TL;DR

This paper tackles a critical overfitting problem in offline reinforcement learning (RL) where policies learn to avoid low-reward situations by manipulating importance weights, leading to overly optimistic performance estimates. They introduce a novel algorithm, POELA, that constrains policies to consider actions observed in similar states, effectively preventing this "propensity overfitting" and achieving better real-world performance.

What Problem Does This Paper Address?

Offline RL is crucial for applications where online learning is expensive or risky, like healthcare. A common approach is to use importance sampling (IS) to estimate the value of a new policy from historical data. However, when optimizing policies using IS, a phenomenon called "overfitting to importance weights" can occur. This means the optimization objective can be tricked into favoring policies that strategically avoid states or actions that have low observed rewards in the dataset, even if those states are unavoidable in reality. This leads to policies that appear to perform very well in simulations but fail in practice. The paper identifies this specific type of overfitting, which is distinct from standard supervised learning overfitting, and proposes a method to mitigate it.

Methods

The core of the paper's contribution is the Policy Optimization with Eligible Actions (POELA) algorithm. The key idea is to introduce a constraint on the policy class. Instead of allowing policies to take any action, POELA restricts them to actions that have been observed in similar states within the dataset. This is achieved by defining an "eligible action set" for each state, based on a neighborhood defined by a distance metric (e.g., Euclidean distance) and a radius $\delta$.

The algorithm then optimizes a policy that adheres to these eligible action constraints. This constraint ensures that the importance weights are lower-bounded within a local neighborhood of observed states, preventing the policy from completely avoiding certain states or actions. The paper provides theoretical justifications for why this constraint helps to avoid overfitting and demonstrates its effectiveness through experiments.

Datasets

The paper uses two primary datasets for evaluation:

  • Healthcare-inspired simulator: A simulator for low-grade glioma (LGG) tumor growth inhibition, which is a non-Markovian environment.
  • Real-world ICU sepsis treatment dataset: A dataset extracted from the MIMIC-III database, containing patient data and treatment logs.

Key Insights and Ideas

  • "Propensity Overfitting": The paper clearly articulates a unique overfitting problem in offline RL where policies learn to "game" the importance sampling objective by avoiding low-reward states, leading to inflated performance estimates. This is not just about fitting the data, but about exploiting the estimation method itself.
  • Eligible Actions as a Solution: The proposed "eligible actions" constraint is a clever way to enforce a form of "local overlap" with the observed data. By only allowing actions seen in similar contexts, the algorithm prevents the policy from making drastic, unobserved choices that could lead to overfitting.
  • Non-Markovian Settings are Key: The problem is particularly relevant in non-Markovian settings, where states don't fully capture the future dynamics, making it harder to rely on traditional model-based or value-based methods that often assume the Markov property. POELA's direct policy search approach is well-suited for these scenarios.
  • Balancing Exploration and Exploitation: The eligible action constraint can be seen as a way to balance the desire to find better policies (exploration) with the need to stay grounded in the observed data (exploitation), preventing the policy from straying too far into unobserved, potentially misleading territory.

Key Results

  • Improved Performance: POELA consistently outperformed state-of-the-art offline RL algorithms (including norm-POEM, BCQ, and PQL) in both the TGI simulator and the MIMIC-III sepsis dataset.
  • Reduced Overfitting: The paper demonstrates that POELA leads to significantly less overfitting, as evidenced by a smaller gap between the estimated policy value (ÛSNTIS) and the true policy value (Test vπ) on validation sets. For instance, in the non-MDP TGI simulator, POELA showed a much smaller difference ( -1.26 ± 1.92) compared to PO-CRM (15.57 ± 13.07).
  • Higher Effective Sample Size (ESS): In the MIMIC-III dataset, POELA achieved a substantially higher effective sample size (528.18) compared to other methods, indicating better overlap with the test data and more reliable performance estimates.
  • Medically Reasonable Actions: In the sepsis experiment, POELA's learned policies for high-risk patients were more aligned with clinician decisions, suggesting they are more medically reasonable than those learned by other methods that tended to avoid high-risk contexts.

Limitations and Future Work

  • Hyperparameter Sensitivity: The performance of POELA is influenced by the choice of the eligible action constraint parameter $\delta$. The paper shows that an intermediate value of $\delta$ often yields the best results, suggesting a trade-off between policy expressivity and constraint effectiveness.
  • Future Work on Behavior Policy Learning: The authors suggest that future work could explore learning an empirical behavior policy to guide the policy search, which might offer additional benefits, though they note subtleties in this approach for large continuous spaces.
  • Clinical Deployment: The authors emphasize that while promising, further investigation and collaboration with clinicians are essential before deploying such algorithms in high-stakes clinical settings.

Summary updated

Abstract

Offline policy optimization has a critical impact on many real-world decision-making problems, as online learning is costly and concerning in many applications. Importance sampling and its variants are a widely used type of estimator in offline policy evaluation, which can be helpful to remove assumptions on the chosen function approximations used to represent value functions and process models. In this paper, we identify an important overfitting phenomenon in optimizing the importance weighted return, and propose an algorithm to avoid this overfitting. We provide a theoretical justification of the proposed algorithm through a better per-state-neighborhood normalization condition and show the limitation of previous attempts to this approach through an illustrative example. We further test our proposed method in a healthcare-inspired simulator and a logged dataset collected from real hospitals. These experiments show the proposed method with less overfitting and better test performance compared with state-of-the-art batch reinforcement learning algorithms.
BibTeX
@inproceedings{liu2022avoiding,
  author = {Yao Liu and Emma Brunskill},
  title = {{Avoiding Overfitting to the Importance Weights in Offline Policy Optimization}},
  booktitle = {International Conference on Learning Representations},
  year = {2022},
  url = {https://openreview.net/forum?id=dLTXoSIcrik},
}
Figure