Scaling Offline RL via Efficient and Expressive Shortcut Models

Nicolas Espinosa-Dice, Yiyi Zhang, Yiding Chen, Bradley Guo, Owen Oertell, Gokul Swamy, Kiante Brantley, Wen Sun

TL;DR

Scalable Offline Reinforcement Learning (SORL) introduces "shortcut models" to offline RL, enabling highly expressive generative policies to be trained via an efficient, one-stage procedure. By incorporating self-consistency, SORL allows agents to dynamically scale their compute budget at test time—either sequentially by adjusting denoising steps or in parallel via best-of-$N$ sampling with a Q-function verifier—to significantly boost performance without retraining.

What Problem Does This Paper Address?

Offline reinforcement learning (RL) trains agents using static, pre-collected datasets without any online exploration. Because these datasets are often highly diverse and multi-modal (containing many different ways to solve a task), standard Gaussian policies struggle to model them accurately. Expressive generative approaches like diffusion and flow matching models handle multi-modal data beautifully, but they come with a catch: their iterative noise-sampling process makes policy optimization incredibly difficult.

Existing methods either require backpropagation through dozens of sequential time steps (which is computationally prohibitive) or rely on complex two-stage distillation processes that compound errors. Furthermore, once these models are trained, their inference compute budget is locked in; they cannot easily trade extra test-time compute for higher precision, a capability that has recently revolutionized large language models.

Methods

The authors introduce SORL, which replaces standard diffusion or flow models with Shortcut Models. Instead of predicting small, incremental steps along a generation trajectory, a shortcut model $s(z_t, t, h)$ is conditioned on both the current timestep $t$ and a variable step size $h$, learning to take larger "jumps" across the trajectory.

SORL uses a simple, unified one-stage training objective consisting of three components:

  1. Q-Loss (Policy Optimization): Maximizes the expected return by backpropagating through a small, variable number of sampling steps ($M^{\text{BTT}}$).
  2. Flow Matching Loss (Data Regularization): Ensures that at the smallest step size, the model accurately recovers the offline data distribution.
  3. Self-Consistency Loss: Enforces that one large jump of size $2h$ yields the exact same result as two consecutive smaller jumps of size $h$.

At test time, this self-consistency property unlocks two types of Inference-Time Scaling:

  • Sequential Scaling: The agent can dynamically choose how many inference steps ($M^{\text{inf}}$) to take based on its current computational budget, even generalizing to more steps than it experienced during training.
  • Parallel Scaling: The agent samples $N$ candidate actions independently and uses the learned Q-function as a verifier to select the action with the highest predicted value (Best-of-$N$ sampling).

Datasets

SORL is evaluated on OGBench, a comprehensive benchmark suite designed for offline goal-conditioned and reward-maximizing RL. The authors test their method across 8 distinct robotics locomotion and manipulation environments (such as antmaze-large, humanoidmaze-large, antsoccer-arena, cube-single, and scene-play), comprising a total of 40 unique tasks.

Key Insights and Ideas

  • Decoupling Training and Inference Compute: Traditional generative policies require the same number of denoising steps during both training and inference. SORL breaks this link. You can train the model efficiently with very few backpropagation steps (e.g., 1 to 4) and still scale up to more steps at test time for superior precision.
  • Test-Time Compute as a Substitute for Training: The authors show that a smaller training budget can be compensated for by allocating more compute at inference time. If an agent has limited time to train but ample time to "think" during deployment, sequential and parallel scaling can recover the performance of a heavily trained model.
  • Theoretical Guarantees on Data Drift: The authors prove that minimizing SORL's flow-matching and self-consistency losses strictly bounds the 2-Wasserstein distance between the generated policy and the offline dataset. This provides a mathematical guarantee against dangerous out-of-distribution actions.

Key Results

SORL achieves state-of-the-art performance, outperforming 10 strong baselines (including Gaussian, diffusion, and flow-based methods) on 5 out of the 8 OGBench environment categories.

SORL's Overall Performance

As shown in the sequential scaling evaluations, increasing the number of inference steps consistently drives up success rates across various environments:

Figure 1: SORL’s Sequential Scaling. For a fixed training budget, SORL generally improves per- formance with greater test-time compute. We fix a training budget of discretization steps and backpropagation steps through time (M disc = M BTT = 8) and vary the inference budget via the number of inference steps M inf. The performance is averaged over 8 seeds for each task, with 5 tasks per environment, and standard deviations reported.

For instance, in the complex scene-play environment, moving from 1 inference step to 8 inference steps boosts the performance score from 33 to 58. Similarly, combining sequential scaling with parallel Best-of-$N$ sampling allows models trained with low compute budgets ($M^{\text{BTT}} = 1$ or $2$) to match or exceed the performance of models trained with higher budgets.

Limitations and Future Work

  • Dependency on Q-Function Accuracy: Parallel scaling relies entirely on the learned Q-function acting as a reliable verifier. If the Q-function is inaccurate or suffers from overestimation errors on out-of-distribution actions, Best-of-$N$ filtering can select sub-optimal actions.
  • Training Runtime: Although SORL simplifies the pipeline by avoiding two-stage distillation, its multi-component loss function results in a longer training runtime per epoch compared to the fastest one-step flow baselines (like FQL).
  • Lack of Online Exploration: SORL is strictly an offline RL algorithm and currently lacks a principled exploration strategy required for optimal performance if transitioned into an online environment.

The authors suggest that future work should explore adaptive test-time scaling, where the model dynamically decides how many inference steps to allocate per state based on the gradient of the Q-function.

Summary updated

Abstract

Diffusion and flow models have emerged as powerful generative approaches capable of modeling diverse and multimodal behavior. However, applying these models to offline reinforcement learning (RL) remains challenging due to the iterative nature of their noise sampling processes, making policy optimization difficult. In this paper, we introduce Scalable Offline Reinforcement Learning (SORL), a new offline RL algorithm that leverages shortcut models - a novel class of generative models - to scale both training and inference. SORL's policy can capture complex data distributions and can be trained simply and efficiently in a one-stage training procedure. At test time, SORL introduces both sequential and parallel inference scaling by using the learned Q-function as a verifier. We demonstrate that SORL achieves strong performance across a range of offline RL tasks and exhibits positive scaling behavior with increased test-time compute. We release the code at nico-espinosadice.github.io/projects/sorl.
BibTeX
@inproceedings{espinosadice2025scaling,
  author = {Nicolas Espinosa-Dice and Yiyi Zhang and Yiding Chen and Bradley Guo and Owen Oertell and Gokul Swamy and Kiante Brantley and Wen Sun},
  title = {{Scaling Offline RL via Efficient and Expressive Shortcut Models}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2025},
  url = {https://openreview.net/forum?id=uVarpp7fhU},
}
Figure