TL;DR
Scalable Offline Reinforcement Learning (SORL) introduces "shortcut models" to offline RL, enabling highly expressive generative policies to be trained via an efficient, one-stage procedure. By incorporating self-consistency, SORL allows agents to dynamically scale their compute budget at test time—either sequentially by adjusting denoising steps or in parallel via best-of-$N$ sampling with a Q-function verifier—to significantly boost performance without retraining.
What Problem Does This Paper Address?
Offline reinforcement learning (RL) trains agents using static, pre-collected datasets without any online exploration. Because these datasets are often highly diverse and multi-modal (containing many different ways to solve a task), standard Gaussian policies struggle to model them accurately. Expressive generative approaches like diffusion and flow matching models handle multi-modal data beautifully, but they come with a catch: their iterative noise-sampling process makes policy optimization incredibly difficult.
Existing methods either require backpropagation through dozens of sequential time steps (which is computationally prohibitive) or rely on complex two-stage distillation processes that compound errors. Furthermore, once these models are trained, their inference compute budget is locked in; they cannot easily trade extra test-time compute for higher precision, a capability that has recently revolutionized large language models.
Methods
The authors introduce SORL, which replaces standard diffusion or flow models with Shortcut Models. Instead of predicting small, incremental steps along a generation trajectory, a shortcut model $s(z_t, t, h)$ is conditioned on both the current timestep $t$ and a variable step size $h$, learning to take larger "jumps" across the trajectory.
SORL uses a simple, unified one-stage training objective consisting of three components:
- Q-Loss (Policy Optimization): Maximizes the expected return by backpropagating through a small, variable number of sampling steps ($M^{\text{BTT}}$).
- Flow Matching Loss (Data Regularization): Ensures that at the smallest step size, the model accurately recovers the offline data distribution.
- Self-Consistency Loss: Enforces that one large jump of size $2h$ yields the exact same result as two consecutive smaller jumps of size $h$.
At test time, this self-consistency property unlocks two types of Inference-Time Scaling:
- Sequential Scaling: The agent can dynamically choose how many inference steps ($M^{\text{inf}}$) to take based on its current computational budget, even generalizing to more steps than it experienced during training.
- Parallel Scaling: The agent samples $N$ candidate actions independently and uses the learned Q-function as a verifier to select the action with the highest predicted value (Best-of-$N$ sampling).
Datasets
SORL is evaluated on OGBench, a comprehensive benchmark suite designed for offline goal-conditioned and reward-maximizing RL. The authors test their method across 8 distinct robotics locomotion and manipulation environments (such as antmaze-large, humanoidmaze-large, antsoccer-arena, cube-single, and scene-play), comprising a total of 40 unique tasks.
Key Insights and Ideas
- Decoupling Training and Inference Compute: Traditional generative policies require the same number of denoising steps during both training and inference. SORL breaks this link. You can train the model efficiently with very few backpropagation steps (e.g., 1 to 4) and still scale up to more steps at test time for superior precision.
- Test-Time Compute as a Substitute for Training: The authors show that a smaller training budget can be compensated for by allocating more compute at inference time. If an agent has limited time to train but ample time to "think" during deployment, sequential and parallel scaling can recover the performance of a heavily trained model.
- Theoretical Guarantees on Data Drift: The authors prove that minimizing SORL's flow-matching and self-consistency losses strictly bounds the 2-Wasserstein distance between the generated policy and the offline dataset. This provides a mathematical guarantee against dangerous out-of-distribution actions.
Key Results
SORL achieves state-of-the-art performance, outperforming 10 strong baselines (including Gaussian, diffusion, and flow-based methods) on 5 out of the 8 OGBench environment categories.
As shown in the sequential scaling evaluations, increasing the number of inference steps consistently drives up success rates across various environments:
For instance, in the complex scene-play environment, moving from 1 inference step to 8 inference steps boosts the performance score from 33 to 58. Similarly, combining sequential scaling with parallel Best-of-$N$ sampling allows models trained with low compute budgets ($M^{\text{BTT}} = 1$ or $2$) to match or exceed the performance of models trained with higher budgets.
Limitations and Future Work
- Dependency on Q-Function Accuracy: Parallel scaling relies entirely on the learned Q-function acting as a reliable verifier. If the Q-function is inaccurate or suffers from overestimation errors on out-of-distribution actions, Best-of-$N$ filtering can select sub-optimal actions.
- Training Runtime: Although SORL simplifies the pipeline by avoiding two-stage distillation, its multi-component loss function results in a longer training runtime per epoch compared to the fastest one-step flow baselines (like FQL).
- Lack of Online Exploration: SORL is strictly an offline RL algorithm and currently lacks a principled exploration strategy required for optimal performance if transitioned into an online environment.
The authors suggest that future work should explore adaptive test-time scaling, where the model dynamically decides how many inference steps to allocate per state based on the gradient of the Q-function.
Summary updated