A Minimalist Approach to Offline Reinforcement Learning

Scott Fujimoto, Shixiang Shane Gu

TL;DR

This paper introduces TD3+BC, a remarkably simple offline reinforcement learning algorithm that achieves state-of-the-art performance by adding just a behavior cloning term and state normalization to the standard online TD3 algorithm. By stripping away the complexity of generative models and intricate regularizers, it matches the performance of far more complex methods while reducing training time by over 80%.

What Problem Does This Paper Address?

Reinforcement Learning (RL) is traditionally an online process: an agent learns by interacting with an environment. Offline RL (or batch RL) changes the game by requiring the agent to learn from a fixed, pre-collected dataset without any new interaction.

The fundamental challenge here is extrapolation error. When an agent evaluates an action not present in the dataset (out-of-distribution), it often wildly overestimates its value. During policy improvement, the agent then gravitates toward these "fake" high-value actions, leading to catastrophic failure.

To combat this, recent state-of-the-art (SOTA) algorithms have become increasingly complex, incorporating generative models, complex divergence penalties, and numerous "implementation-level" adjustments (like specific network architectures or reward bonuses). The authors argue that this complexity makes these algorithms difficult to tune, reproduce, and understand, as it's unclear which specific component is actually driving performance.

Methods

The authors propose a "minimalist" approach. They take the standard TD3 (Twin Delayed Deep Deterministic policy gradient) algorithm and make only two changes:

  1. Behavior Cloning (BC) Regularization: They modify the policy update to include a simple mean-squared error term that penalizes the learned policy $\pi(s)$ for deviating from the actions $a$ found in the dataset: $$\pi = \text{argmax}_{\pi} \mathbb{E}_{(s,a) \sim \mathcal{D}} [\lambda Q(s, \pi(s)) - (\pi(s) - a)^2]$$ Here, $\lambda$ is a hyperparameter that balances maximizing the Q-value (RL) with staying close to the data (BC). To make $\lambda$ robust across different reward scales, they use a heuristic: $\lambda = \frac{\alpha}{\frac{1}{N} \sum |Q(s_i, a_i)|}$, where $\alpha$ is a fixed constant.

  2. State Normalization: They normalize the state features in the dataset to have a mean of 0 and a standard deviation of 1. While a common trick in online RL, they find it provides a non-trivial performance boost in the offline setting.

Importantly, these are the only changes. No generative models, no complex target updates, and no architecture modifications.

Datasets

The algorithm was evaluated on the D4RL benchmark, a standard suite for offline RL consisting of continuous control tasks (MuJoCo environments like HalfCheetah, Hopper, and Walker2d). The benchmark includes datasets collected by different policies:

  • Random: Data from a completely random policy.
  • Medium: Data from a partially trained policy.
  • Medium-Replay: The entire replay buffer of a policy trained to "medium" performance.
  • Medium-Expert: A mix of medium-level and expert-level data.
  • Expert: Data from a fully trained expert policy.

Key Insights and Ideas

  • Complexity is Overrated: The most striking finding is that a simple BC term is just as effective as the complex machinery used in other SOTA methods. This suggests the community may be over-emphasizing algorithmic novelty at the expense of simplicity.
  • The "Hidden" Cost of Implementation: The authors show that many previous SOTA methods rely heavily on non-algorithmic implementation details. As seen in Figure 1, removing these "tricks" from algorithms like CQL or Fisher-BRC leads to massive performance drops. Performance drop when removing implementation adjustments
  • Shared Instability: A key "aha moment" is that all offline RL algorithms—including the simple TD3+BC—suffer from high variance in performance across different evaluation episodes and training steps. This is a stark contrast to online RL, which typically converges to stable policies. This suggests that distributional shift remains a fundamental, unsolved problem for the entire field. High episodic variance in offline RL
Figure 1: Percent difference of performance of offline RL algorithms and their simplified versions which remove implementation adjustments to their underlying algorithm. HC = HalfCheetah, Hop = Hopper, W = Walker, r = random, m = medium, mr = medium-replay, me = medium-expert, e = expert. Huge drops in performances show that the implementation complexities are crucial for achieving the best results in these prior algorithms.
Figure 2: Percent difference of the worst episode during the 10 evaluation episodes at the last evaluation. This measures the deviations in performance at single point in time. HC = HalfCheetah, Hop = Hopper, W = Walker, r = random, m = medium, mr = medium-replay, me = medium-expert, e = expert. While online algorithms (TD3) typically have small episode variances per trained policy (as they should at convergence), all offline algorithms have surprisingly high episodic variances for trained policies.

Key Results

  • SOTA Performance: TD3+BC matches or exceeds the performance of complex baselines like CQL and Fisher-BRC across almost all D4RL tasks.
  • Massive Speedup: Because it removes the computational overhead of generative models and complex sampling, TD3+BC is significantly faster. For 1 million training steps, TD3+BC takes only 39 minutes, compared to over 4 hours for CQL. Run time comparison
  • Learning Efficiency: As shown in the learning curves, TD3+BC reaches peak performance quickly and maintains it, proving that minimalist changes can be highly effective. Learning curves comparison
Run time comparison
Figure 4: Learning curves comparing the performance of TD3+BC against offline RL baselines in the D4RL datasets. Curves are averaged over 5 seeds, with the shaded area representing the standard deviation across seeds. TD3+BC exhibits a similar learning speed and final performance as the state-of-the-art Fisher-BRC, without the need of pre-training a generative model.

Limitations and Future Work

  • Random Data: TD3+BC performs relatively poorly on purely random datasets compared to some other methods. The authors suggest this is because the simple BC term might be too restrictive when the data is highly diverse and low-quality.
  • Policy Instability: The high variance in trained policies (Figures 2 and 3 in the paper) is a significant hurdle for real-world deployment, where reliability is paramount. The authors call on the community to address this "neglected problem."
  • Hyperparameter Tuning: While simpler, the algorithm still requires tuning the $\alpha$ parameter, though the authors found a single value ($\alpha=2.5$) worked well across most tasks.

In conclusion, this paper serves as a powerful reminder that in the quest for performance, we shouldn't overlook the most straightforward solutions. TD3+BC provides a strong, efficient, and easy-to-implement baseline for future offline RL research.

Summary updated

Abstract

Offline reinforcement learning (RL) defines the task of learning from a fixed batch of data. Due to errors in value estimation from out-of-distribution actions, most offline RL algorithms take the approach of constraining or regularizing the policy with the actions contained in the dataset. Built on pre-existing RL algorithms, modifications to make an RL algorithm work offline comes at the cost of additional complexity. Offline RL algorithms introduce new hyperparameters and often leverage secondary components such as generative models, while adjusting the underlying RL algorithm. In this paper we aim to make a deep RL algorithm work while making minimal changes. We find that we can match the performance of state-of-the-art offline RL algorithms by simply adding a behavior cloning term to the policy update of an online RL algorithm and normalizing the data. The resulting algorithm is a simple to implement and tune baseline, while more than halving the overall run time by removing the additional computational overhead of previous methods.
BibTeX
@inproceedings{fujimoto2021minimalist,
  author = {Scott Fujimoto and Shixiang Shane Gu},
  title = {{A Minimalist Approach to Offline Reinforcement Learning}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2021},
  url = {https://openreview.net/forum?id=7AdUxIuOB9},
}
Figure