Effective Offline Reinforcement Learning via Conservative State Value Estimation

Liting Chen, Jie Yan, Lu Wang, Qingwei Lin, Dongmei Zhang, Saravan Rajmohan, Thomas MoscibrodaICLR 2023 · OpenReview

TL;DR

This paper introduces CSVE, a novel offline reinforcement learning (RL) approach that tackles the critical issue of value overestimation by directly penalizing state value functions on out-of-distribution (OOD) states. CSVE demonstrates improved performance over existing conservative Q-learning methods and achieves competitive results with state-of-the-art algorithms on standard benchmarks.

What Problem Does This Paper Address?

Offline reinforcement learning (RL) aims to learn effective policies from pre-collected datasets without further interaction with the environment. This is crucial for real-world applications where online exploration might be risky or costly. However, a major hurdle in offline RL is value overestimation. This occurs when the learned policy encounters states or actions not well-represented in the dataset (out-of-distribution or OOD). The value function, when naively estimated, can assign overly optimistic values to these OOD scenarios, leading to poor policy performance or even learning failure. Existing methods like Conservative Q-Learning (CQL) address this by penalizing the Q-value function for OOD actions. While effective, these methods can sometimes be overly pessimistic. This paper seeks to find a more effective way to conservatively estimate values, leading to better policies.

Methods

The core innovation of CSVE lies in its approach to conservative value estimation. Instead of penalizing Q-values (state-action values) for OOD actions, CSVE directly penalizes the V-value function (state value) for OOD states. This is achieved through a modified Bellman update that incorporates a penalty term.

The paper proposes an actor-critic framework:

  • Conservative Value Estimation (Critic): The critic learns a conservative V-function by minimizing a loss that penalizes deviations from the true value on OOD states. This involves sampling states "around" the dataset, effectively creating a "soft river" of conservative values.
  • Advantage Weighted Policy Updates (Actor): The actor uses the learned conservative V-values (specifically, estimated advantages) to update the policy. This method, inspired by Advantage Weighted Regression (AWR), implicitly constrains the learned policy to stay close to the behavior policy, preventing excessive extrapolation.

The authors also leverage a dynamics model for efficient sampling of OOD states that are reachable from the dataset, avoiding the need for a perfect model or extensive data coverage.

Datasets

The paper evaluates CSVE on standard offline RL benchmarks:

  • Gym continuous control tasks: Including HalfCheetah, Hopper, and Walker2D, with datasets collected from various policies (random, medium, medium-replay, medium-expert, expert).
  • Adroit tasks: Including Pen, Hammer, Door, and Relocate, with datasets collected from human, cloned, and expert policies.

These datasets are part of the D4RL (Datasets for Deep Data-Driven Reinforcement Learning) benchmark.

Key Insights and Ideas

  • Penalizing V vs. Q: The paper argues that penalizing V-values for OOD states is more effective and flexible than penalizing Q-values. This is because V-value conservatism provides a lower bound on the expected value, which is less restrictive than state-wise Q-value conservatism, allowing for more adventurous policies.
  • "Soft River" of Conservatism: By sampling states "around" the dataset and penalizing their values, CSVE creates a conservative buffer zone, guiding the policy away from truly OOD regions without being overly restrictive.
  • Implicit Policy Constraints: The use of advantage-weighted updates naturally constrains the learned policy, ensuring it doesn't stray too far from the data distribution, which is crucial for safe and effective offline learning.
  • Model-Assisted Exploration: Using a dynamics model to sample OOD states that are "reachable" from the dataset is an efficient way to guide exploration without requiring a perfect model or extensive data.

Key Results

  • Improved Performance: CSVE consistently outperforms CQL on 11 out of 15 Gym control tasks and shows similar performance on others. It also demonstrates better performance than AWAC on 9 out of 15 tasks.
  • Competitive with SOTA: CSVE is shown to be strongly competitive with state-of-the-art offline RL algorithms, often achieving comparable results to methods like PBRL, which require significantly more computational resources.
  • Robustness to Model Bias: Experiments suggest that CSVE is relatively insensitive to biases in the learned dynamics model, a common issue in model-based RL.
  • Effectiveness on Diverse Datasets: The method shows particular advantage on datasets generated by random or sub-optimal policies, where OOD issues are more pronounced.

Limitations and Future Work

  • Hyperparameter Sensitivity: While the paper explores hyperparameter effects, finding the optimal settings (e.g., for the penalty factor $\alpha$ and exploration bonus $\lambda$) can be task-specific.
  • Future Work: The authors suggest exploring more powerful algorithms based on conservative state value estimation and investigating further applications. They also highlight the potential for more adventurous policies by bounding expected values rather than state-wise values, which CSVE's approach enables.

Summary updated

Abstract

Offline RL promises to learn effective policies from static experience datasets without further interaction, which expect to perform well in the online environment. However, it faces up to a major challenge of value over-estimation introduced by the distributional drift between the dataset and the current learned policy, which leads to learning failure in practice. The common approach is to add a penalty term to reward or value estimation in the Bellman iterations, which has given rise to a number of successful algorithms such as CQL. Meanwhile, to avoid extrapolation on unseen states, existing methods focus on conservative Q-function estimation. In this paper, we propose CSVE, a new approach that directly imposes penalty on out-of-distribution states. We prove that for the evaluated policy, our conservative state value estimation satisfies: (1) over the state distribution that samples penalizing states, it lower bounds the true values in expectation, and (2) over the marginal state distribution of data, it is no more than the true values in expectation plus a constant decided by sampling error. Further, we develop a practical actor-critic algorithm in which the critic does the conservative value estimation by additionally sampling and penalizing the states 'around' the dataset, while the actor applies advantage weighted updates to improve the policy. We evaluate in classic continual control tasks of D4RL, showing that our method performs better than the conservative Q-function learning methods (e.g., CQL) and is strongly competitive among recent SOTA methods.
BibTeX
@inproceedings{chen2023effective,
  author = {Liting Chen and Jie Yan and Lu Wang and Qingwei Lin and Dongmei Zhang and Saravan Rajmohan and Thomas Moscibroda},
  title = {{Effective Offline Reinforcement Learning via Conservative State Value Estimation}},
  booktitle = {International Conference on Learning Representations},
  year = {2023},
  url = {https://openreview.net/forum?id=aySB6rDo0z},
}
Figure