RvS: What is Essential for Offline RL via Supervised Learning?

Scott Emmons, Benjamin Eysenbach, Ilya Kostrikov, Sergey Levine

Introduction

Offline Reinforcement Learning (RL) doesn't have to be complicated. This paper demonstrates that a simple two-layer neural network trained with standard supervised learning—no temporal difference (TD) learning or complex Transformers required—can match or exceed state-of-the-art performance. The secret sauce isn't a fancy architecture, but rather choosing the right information to condition on (goals vs. rewards) and carefully tuning model capacity.

The Problem with Traditional Offline RL

Traditional offline RL—learning from a fixed dataset without further environment interaction—is dominated by value-based methods like TD-learning. While powerful, these methods are notoriously "fussy": they require complex stability tricks and delicate hyperparameter tuning to prevent value estimates from exploding.

Recently, a new family of methods called RL via Supervised Learning (RvS) emerged. These treat RL as a conditional imitation learning problem: "If I want to achieve [Goal X] or [Reward Y], what action should I take?" However, prior work in this area often used high-capacity sequence models (like Transformers in the Decision Transformer) or complex weighting schemes. The authors of this paper ask a fundamental question: What is actually essential for RvS to work? Do we really need Transformers and complex weighting, or can we get away with something much simpler?

The RvS Framework: Simplicity is Key

The authors propose a unified framework for RvS that boils the process down to its barest essentials, avoiding sequence modeling and advantage weighting entirely.

The RvS framework: from replay buffer to a simple MLP architecture.
  1. Hindsight Relabeling: Take a trajectory from the dataset. Pick a future state as a "goal" or the total sum of future rewards as a "reward-to-go."
  2. Simple Architecture: Use a standard multi-layer perceptron (MLP) with just two hidden layers.
  3. Supervised Learning: Train the MLP to predict the action taken in the dataset, conditioned on the current state and the chosen outcome (goal or reward).
  4. Two Flavors: They evaluate RvS-G (conditioning on goals) and RvS-R (conditioning on rewards).

Key Insights: What Makes RvS Tick?

Through extensive evaluation across standard offline RL benchmarks (GCSL Suite, D4RL Gym Locomotion, Franka Kitchen, and AntMaze), the authors uncovered several critical insights:

  • The "Simplicity" Revelation: A vanilla MLP can perform as well as a Transformer. This suggests that for many offline RL tasks, modeling the long-term history of states (sequence modeling) isn't as important as simply knowing the current state and the intended target.
  • Conditioning is Domain-Specific: There is no "one size fits all." RvS-G (Goals) is superior for navigation and manipulation (AntMaze, Kitchen) because it helps the model "stitch" together sub-trajectories spatially. RvS-R (Rewards) is better for locomotion (Gym) where the "goal" is an abstract velocity rather than a specific coordinate.
  • Capacity vs. Regularization: RvS models need more capacity (wider layers) than standard RL agents because they have to represent many different behaviors (optimal and suboptimal) simultaneously. However, this high capacity requires careful regularization (like Dropout) to prevent overfitting, especially on small datasets like Franka Kitchen.

Key Results

Despite its simplicity, the RvS approach yields highly competitive results across the board.

Performance comparison across D4RL Kitchen, Gym, and AntMaze suites.
  • State-of-the-Art Performance: RvS-G achieved state-of-the-art results on the AntMaze and Franka Kitchen benchmarks, outperforming much more complex TD-learning algorithms.
  • Matching Transformers: On the D4RL Gym tasks, the simple RvS-R MLP matched the performance of the Decision Transformer.
  • Spatial Stitching: The authors showed that RvS-G can effectively perform "subtrajectory stitching"—combining parts of different unsuccessful paths to find a successful one—without needing the explicit temporal backups used in value-based RL.

Limitations and Future Work

While RvS is a powerful and simple baseline, it is not without its challenges:

  • The Limits of RvS on Random Data: While RvS is excellent on "play" or "expert" data, it struggles on purely random data. In these cases, TD-learning methods (like CQL) still hold the crown because they can better synthesize a policy from noisy, non-directed transitions.
  • The Tuning Trap: One major hurdle is that validation loss is a poor proxy for actual performance. A model might have a low error in predicting actions but still fail to reach the goal in practice. Finding a reliable offline metric for hyperparameter tuning remains an open problem.
  • Manual Conditioning: Currently, a human must decide whether to use goals or rewards and what the "target" value should be at test time. Automating this choice is a key direction for future research.

Summary updated

Abstract

Recent work has shown that supervised learning alone, without temporal difference (TD) learning, can be remarkably effective for offline RL. When does this hold true, and which algorithmic components are necessary? Through extensive experiments, we boil supervised learning for offline RL down to its essential elements. In every environment suite we consider, simply maximizing likelihood with a two-layer feedforward MLP is competitive with state-of-the-art results of substantially more complex methods based on TD learning or sequence modeling with Transformers. Carefully choosing model capacity (e.g., via regularization or architecture) and choosing which information to condition on (e.g., goals or rewards) are critical for performance. These insights serve as a field guide for practitioners doing Reinforcement Learning via Supervised Learning (which we coin RvS learning). They also probe the limits of existing RvS methods, which are comparatively weak on random data, and suggest a number of open problems.
BibTeX
@inproceedings{emmons2022rvs,
  author = {Scott Emmons and Benjamin Eysenbach and Ilya Kostrikov and Sergey Levine},
  title = {{RvS: What is Essential for Offline RL via Supervised Learning?}},
  booktitle = {International Conference on Learning Representations},
  year = {2022},
  url = {https://openreview.net/forum?id=z5CaNoV2xke},
}
Figure