Continuous Doubly Constrained Batch Reinforcement Learning

Rasool Fakoor, Jonas Mueller, Kavosh Asadi, Pratik Chaudhari, Alex Smola

Introduction

In standard online Reinforcement Learning (RL), agents learn by interacting with an environment, trying out actions, and observing the results. However, in many real-world scenarios—such as healthcare or robotics—trial-and-error is too expensive or dangerous. Batch (or offline) RL attempts to learn optimal policies strictly from a fixed, pre-collected dataset without any new environment interaction.

This paper introduces Continuous Doubly Constrained Batch RL (CDC), a novel algorithm for offline reinforcement learning that prevents Q-value estimates from exploding when evaluating unseen actions. By applying two simple, complementary penalties—one to constrain Q-value overestimation and another to keep the learned policy close to the behavior policy—CDC achieves state-of-the-art performance on continuous control benchmarks.

The Challenge of Batch RL

The fatal flaw in batch RL is extrapolation-inflated overestimation. Because a fixed dataset only covers a small fraction of all possible state-action pairs, the neural networks estimating the value of actions (Q-networks) must guess the value of out-of-distribution (OOD) actions.

These guesses are often noisy and overly optimistic. The policy optimizer naturally favors these optimistic OOD actions, and standard temporal difference (TD) updates use these inflated estimates as targets for future updates. This creates a vicious cycle where Q-values for unseen actions explode to absurdly high numbers, leading to the deployment of highly risky, catastrophic policies.

Standard Actor-Critic suffers from exploding Q-values, while CDC maintains stability and achieves high returns.

The CDC Approach

To break this cycle of overestimation, the authors propose an off-policy actor-critic algorithm called CDC, which introduces two novel regularizers:

  1. Q-Value Regularization (The Value Constraint): The authors add a penalty ($\Delta$) to the Q-network update. It specifically penalizes the network if the estimated Q-value for a candidate action (chosen by the current policy) significantly exceeds the Q-value of the action actually observed in the dataset for that same state. Crucially, this penalty only activates to curb dangerous optimism; it does not penalize the network if the candidate action's value is lower than the observed action's value.

  2. Policy Regularization (The Policy Constraint): To prevent the agent from wandering too far into OOD territory in the first place, the authors add an exploration penalty to the policy update. This forces the learned policy to maximize Q-values while remaining close to the unknown behavior policy that generated the data. They achieve this by maximizing the log-likelihood of the dataset actions under the new policy, which acts as an efficient estimate of reverse KL-divergence.

These constraints are applied on top of a standard ensemble of Q-networks, using the lower confidence bound of the ensemble to further account for uncertainty.

Algorithm 1: Continuous Doubly Constrained Batch RL.

Experimental Setup and Key Results

The authors evaluated CDC on 32 continuous control tasks from the D4RL (Datasets for Deep Data-Driven Reinforcement Learning) benchmark. Environments included simulated MuJoCo robotics (HalfCheetah, Hopper, Walker2d), AntMaze, Adroit manipulation tasks, and FrankaKitchen. The datasets varied in quality, ranging from data collected by random policies to expert human demonstrators.

  • State-of-the-Art Returns: Across all 32 D4RL tasks, CDC achieved a total normalized return of 1397, significantly outperforming the next-best baseline (CQL), which scored 1245.
  • Robustness: CDC remained highly competitive regardless of the environment or the quality of the dataset. It showed statistically significant improvements over strong baselines like BEAR, BRAC, BCQ, and SAC.
  • OPE Improvement: In Offline Policy Evaluation (OPE)—the practice of scoring a policy before deploying it—adding the $\Delta$ penalty to Fitted Q Evaluation (FQE) improved the average Pearson correlation with true returns from a dismal 0.01 to 0.37. In 15 of the 32 tasks, standard FQE actually correlated negatively with true returns due to overestimation; the $\Delta$ penalty reduced this failure mode to just 4 tasks.

Key Insights

  • Double constraints are synergistic: Ablation studies reveal that while both constraints help, using them together yields the best performance. The value constraint ($\Delta$) is the heavier lifter, but the policy constraint provides necessary stability.
  • Bounded pessimism is better than infinite pessimism: Unlike some offline RL methods that push OOD Q-values down infinitely (leading to overly conservative agents), CDC's value penalty only activates when candidate actions are valued higher than known actions. Once they are equalized, the penalty drops to zero, allowing the agent to still find improved policies.
  • No need to model the behavior policy: Estimating the exact behavior policy from a dataset is notoriously difficult in continuous action spaces. CDC cleverly bypasses this by using a log-likelihood term that minimizes the reverse KL-divergence directly from the batch data.

Limitations and Future Directions

The authors acknowledge that real-world offline datasets often contain unobserved confounders (variables that affect both the action taken and the outcome, but aren't recorded in the state data). CDC, like most current RL algorithms, assumes the provided state is fully observable. Additionally, while the specific penalties proposed in CDC are simple and highly effective, other mathematical combinations of policy and value regularizers might perform even better and warrant future exploration.

Summary updated

Abstract

Reliant on too many experiments to learn good actions, current Reinforcement Learning (RL) algorithms have limited applicability in real-world settings, which can be too expensive to allow exploration. We propose an algorithm for batch RL, where effective policies are learned using only a fixed offline dataset instead of online interactions with the environment. The limited data in batch RL produces inherent uncertainty in value estimates of states/actions that were insufficiently represented in the training data. This leads to particularly severe extrapolation when our candidate policies diverge from one that generated the data. We propose to mitigate this issue via two straightforward penalties: a policy-constraint to reduce this divergence and a value-constraint that discourages overly optimistic estimates. Over a comprehensive set of $32$ continuous-action batch RL benchmarks, our approach compares favorably to state-of-the-art methods, regardless of how the offline data were collected.
BibTeX
@inproceedings{fakoor2021continuous,
  author = {Rasool Fakoor and Jonas Mueller and Kavosh Asadi and Pratik Chaudhari and Alex Smola},
  title = {{Continuous Doubly Constrained Batch Reinforcement Learning}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2021},
  url = {https://openreview.net/forum?id=RIEqVBFDJTR},
}
Figure