Introduction
Offline Reinforcement Learning (RL) doesn't have to be complicated. This paper demonstrates that a simple two-layer neural network trained with standard supervised learning—no temporal difference (TD) learning or complex Transformers required—can match or exceed state-of-the-art performance. The secret sauce isn't a fancy architecture, but rather choosing the right information to condition on (goals vs. rewards) and carefully tuning model capacity.
The Problem with Traditional Offline RL
Traditional offline RL—learning from a fixed dataset without further environment interaction—is dominated by value-based methods like TD-learning. While powerful, these methods are notoriously "fussy": they require complex stability tricks and delicate hyperparameter tuning to prevent value estimates from exploding.
Recently, a new family of methods called RL via Supervised Learning (RvS) emerged. These treat RL as a conditional imitation learning problem: "If I want to achieve [Goal X] or [Reward Y], what action should I take?" However, prior work in this area often used high-capacity sequence models (like Transformers in the Decision Transformer) or complex weighting schemes. The authors of this paper ask a fundamental question: What is actually essential for RvS to work? Do we really need Transformers and complex weighting, or can we get away with something much simpler?
The RvS Framework: Simplicity is Key
The authors propose a unified framework for RvS that boils the process down to its barest essentials, avoiding sequence modeling and advantage weighting entirely.
- Hindsight Relabeling: Take a trajectory from the dataset. Pick a future state as a "goal" or the total sum of future rewards as a "reward-to-go."
- Simple Architecture: Use a standard multi-layer perceptron (MLP) with just two hidden layers.
- Supervised Learning: Train the MLP to predict the action taken in the dataset, conditioned on the current state and the chosen outcome (goal or reward).
- Two Flavors: They evaluate RvS-G (conditioning on goals) and RvS-R (conditioning on rewards).
Key Insights: What Makes RvS Tick?
Through extensive evaluation across standard offline RL benchmarks (GCSL Suite, D4RL Gym Locomotion, Franka Kitchen, and AntMaze), the authors uncovered several critical insights:
- The "Simplicity" Revelation: A vanilla MLP can perform as well as a Transformer. This suggests that for many offline RL tasks, modeling the long-term history of states (sequence modeling) isn't as important as simply knowing the current state and the intended target.
- Conditioning is Domain-Specific: There is no "one size fits all." RvS-G (Goals) is superior for navigation and manipulation (AntMaze, Kitchen) because it helps the model "stitch" together sub-trajectories spatially. RvS-R (Rewards) is better for locomotion (Gym) where the "goal" is an abstract velocity rather than a specific coordinate.
- Capacity vs. Regularization: RvS models need more capacity (wider layers) than standard RL agents because they have to represent many different behaviors (optimal and suboptimal) simultaneously. However, this high capacity requires careful regularization (like Dropout) to prevent overfitting, especially on small datasets like Franka Kitchen.
Key Results
Despite its simplicity, the RvS approach yields highly competitive results across the board.
- State-of-the-Art Performance: RvS-G achieved state-of-the-art results on the AntMaze and Franka Kitchen benchmarks, outperforming much more complex TD-learning algorithms.
- Matching Transformers: On the D4RL Gym tasks, the simple RvS-R MLP matched the performance of the Decision Transformer.
- Spatial Stitching: The authors showed that RvS-G can effectively perform "subtrajectory stitching"—combining parts of different unsuccessful paths to find a successful one—without needing the explicit temporal backups used in value-based RL.
Limitations and Future Work
While RvS is a powerful and simple baseline, it is not without its challenges:
- The Limits of RvS on Random Data: While RvS is excellent on "play" or "expert" data, it struggles on purely random data. In these cases, TD-learning methods (like CQL) still hold the crown because they can better synthesize a policy from noisy, non-directed transitions.
- The Tuning Trap: One major hurdle is that validation loss is a poor proxy for actual performance. A model might have a low error in predicting actions but still fail to reach the goal in practice. Finding a reliable offline metric for hyperparameter tuning remains an open problem.
- Manual Conditioning: Currently, a human must decide whether to use goals or rewards and what the "target" value should be at test time. Automating this choice is a key direction for future research.
Summary updated