Abstract
Recent work has shown that supervised learning alone, without temporal difference (TD) learning, can be remarkably effective for offline RL. When does this hold true, and which algorithmic components are necessary? Through extensive experiments, we boil supervised learning for offline RL down to its essential elements. In every environment suite we consider, simply maximizing likelihood with a two-layer feedforward MLP is competitive with state-of-the-art results of substantially more complex methods based on TD learning or sequence modeling with Transformers. Carefully choosing model capacity (e.g., via regularization or architecture) and choosing which information to condition on (e.g., goals or rewards) are critical for performance. These insights serve as a field guide for practitioners doing Reinforcement Learning via Supervised Learning (which we coin "RvS learning"). They also probe the limits of existing RvS methods, which are comparatively weak on random data, and suggest a number of open problems.
Introduction
Offline Reinforcement Learning (RL) has traditionally been dominated by Temporal Difference (TD) learning methods. While effective, TD learning is often complex to implement and sensitive to hyperparameters, particularly in the offline setting where exploration is impossible. Recent research has explored treating RL as a supervised learning problem, often utilizing large-scale sequence models like Transformers. This work investigates whether such complexity is truly necessary. We demonstrate that a simple two-layer Multi-Layer Perceptron (MLP) trained via supervised learning—a method we call RvS—can match or even exceed the performance of state-of-the-art TD-learning and Transformer-based methods. Our findings suggest that the keys to success lie not in architectural complexity, but in carefully managing model capacity and selecting appropriate conditioning information.
Methods: RL via Supervised Learning (RvS)
The core of RvS is Conditional Behavior Cloning. Instead of learning a value function, the agent learns to predict actions that lead to specific outcomes observed in the training data.
The RvS Pipeline
The training process involves three main steps:
- Hindsight Relabeling: Every trajectory in the offline dataset is treated as a successful demonstration of reaching its own final state or achieving its own total reward.
- Conditioning: The model is trained to predict an action $a$ given the current state $s$ and a target outcome $\omega$.
- Supervised Training: We maximize the likelihood of the actions in the dataset using a standard cross-entropy or mean-squared error loss, depending on the action space.
We explore two primary variants of conditioning:
- RvS-G (Goals): The model is conditioned on a target state $\omega = g$. This is particularly effective for tasks with clear spatial objectives.
- RvS-R (Rewards): The model is conditioned on a target return-to-go $\omega = R$. This is better suited for tasks where the goal is defined by a continuous performance metric rather than a specific terminal state.
Key Insights
Our extensive experimentation yielded several critical insights for practitioners:
1. Simplicity is Sufficient
A standard two-layer MLP can match the performance of the Decision Transformer and sophisticated TD-learning methods like CQL. This suggests that the "magic" of recent supervised RL methods may come more from the conditioning framework than from the Transformer architecture itself.
2. The Capacity Paradox
While traditional RL agents often use small networks, RvS benefits significantly from higher model capacity. Because the model must learn to associate many different actions with their respective outcomes across the entire dataset (not just the optimal ones), a wider and deeper network is required to capture this complex mapping.
3. Conditioning is Task-Dependent
There is no universal best conditioning variable. RvS-G excels in navigation and manipulation tasks (like AntMaze and Franka Kitchen) because spatial goals provide a strong signal for "stitching" together trajectory segments. In contrast, RvS-R is superior for locomotion tasks (like the D4RL Gym suite) where the objective is maintaining a high-velocity gait.
Key Results
RvS demonstrates remarkable performance across standard benchmarks, often outperforming much more complex algorithms.
- AntMaze: In the challenging "Large-Diverse" AntMaze, RvS-G achieved a score of 36.9, significantly higher than CQL's 14.9. This proves that supervised learning can perform "spatial stitching" without explicit dynamic programming.
- Franka Kitchen: On the "Mixed" task, RvS-G reached a score of 60.3, surpassing CQL's 51.0.
- Gym Locomotion: RvS-R matched the performance of the Decision Transformer across nearly all tasks, proving that a simple MLP is a competitive alternative to sequence models for these environments.
Discussion and Future Work
While RvS is a powerful and simple baseline, it has limitations. It struggles with purely random data where TD-learning methods still hold an advantage in signal extraction. Furthermore, the "oracle" problem—knowing which goal or reward to request at test time—remains a challenge in purely offline settings. Finally, we observed a "validation gap" where standard offline metrics like validation loss do not reliably predict online performance, highlighting the need for better offline model selection techniques.
Summary updated