Abstract
Diffusion and flow models have emerged as powerful generative approaches capable of modeling diverse and multimodal behavior. However, applying these models to offline reinforcement learning (RL) remains challenging due to the iterative nature of their noise sampling processes, making policy optimization difficult. In this paper, we introduce Scalable Offline Reinforcement Learning (SORL), a new offline RL algorithm that leverages shortcut models—a novel class of generative models—to scale both training and inference. SORL's policy can capture complex data distributions and can be trained simply and efficiently in a one-stage training procedure. At test time, SORL introduces both sequential and parallel inference scaling by using the learned $Q$-function as a verifier. We demonstrate that SORL achieves strong performance across a range of offline RL tasks and exhibits positive scaling behavior with increased test-time compute.
Introduction
Offline Reinforcement Learning (RL) involves training agents on fixed datasets without the luxury of online exploration. This paradigm presents two major scaling challenges. First, real-world datasets are often "multi-modal," containing many different ways to solve a task. Standard Gaussian policies are typically too simple to capture this diversity. While Diffusion and Flow Matching models offer the necessary expressivity, they are notoriously difficult to optimize for RL because their iterative nature makes backpropagation through time computationally expensive or requires complex two-stage distillation processes.
Second, there is a need for inference flexibility. Ideally, an agent should be able to act instantly for high-speed tasks, such as autonomous driving, while utilizing extra compute for high-precision tasks, such as robotic surgery. Most existing generative RL models are locked into the specific number of steps they were trained with. SORL (Scalable Offline Reinforcement Learning) addresses these issues by introducing "shortcut models" to offline RL, enabling a single-stage training process for highly expressive, multi-modal policies that can scale performance at test-time without retraining.
Methodology
The Shortcut Model
The core innovation of SORL is the use of Shortcut Models. Instead of learning to take tiny, fixed-size steps to "denoise" an action—as is standard in Diffusion—a shortcut model $s_\theta(z, t, h)$ learns to predict the direction to the next point for any given step size $h$.
Self-Consistency and One-Stage Training
SORL is built upon a Self-Consistency Loss. This property enforces that one large "jump" of size $2h$ must land in the same place as two consecutive smaller "jumps" of size $h$. This allows the model to learn the entire trajectory flow efficiently.
Unlike "distillation" methods that train a slow model and then compress it into a fast one, SORL is a unified model that learns to be both fast and precise simultaneously. The actor's objective integrates three primary terms in a one-stage training procedure:
- Q-Loss: Maximizing the expected return to optimize the RL objective.
- Flow Matching Loss: Ensuring the policy stays close to the offline data distribution (Behavioral Cloning).
- Self-Consistency Loss: Ensuring the model can handle varying step sizes and maintain trajectory integrity.
Theoretical analysis shows that the combination of flow matching and self-consistency acts as a Wasserstein behavioral regularization, mathematically guaranteeing that the policy remains within the safe, known data distribution of the offline set.
Test-Time Scaling
SORL enables agents to trade compute for precision at inference time through two mechanisms:
- Sequential Scaling: Increasing the number of Euler integration steps at inference to reduce discretization error. Surprisingly, SORL can generalize to a higher number of inference steps at test-time than the maximum number of steps used during its training.
- Parallel Scaling (Best-of-N): Sampling $N$ actions independently and using the learned $Q$-function as a "verifier" to select the optimal action. This allows the agent to "look harder" at the available options when compute permits.
Experimental Results
Performance on OGBench
The researchers evaluated SORL on OGBench, a comprehensive benchmark for offline RL consisting of 40 unique tasks across 8 environments, including locomotion (e.g., antmaze, humanoidmaze), manipulation (scene-play, cube-single/double), and agility (antsoccer).
SORL achieved state-of-the-art performance, outperforming 10 established baselines (including FQL, IDQL, and ReBRAC) in 5 out of 8 environment categories. While more expressive than one-step models, SORL maintains a runtime comparable to the fastest flow-based baselines when configured for low-latency inference.
Scaling Behavior
The experiments demonstrated positive scaling curves in environments like antsoccer and scene-play, where performance increased significantly as the number of inference steps grew from 1 to 8. Furthermore, SORL can compensate for a smaller training budget by using more compute at test-time, reaching performance levels equivalent to models trained with significantly higher budgets.
Discussion and Limitations
While SORL offers a powerful framework for scalable RL, it has certain limitations. Parallel scaling (Best-of-N) relies heavily on the accuracy of the learned $Q$-function; if the $Q$-function is biased toward out-of-distribution actions, parallel scaling can degrade performance. Additionally, while the one-stage training is flexible, the self-consistency checks result in a longer training runtime than the simplest one-step distillation models.
As an offline RL algorithm, SORL also lacks a principled exploration strategy for online settings. Future work may investigate adaptive test-time scaling, where the agent dynamically adjusts its computational expenditure based on the perceived difficulty of the current state.
Summary updated