Offline reinforcement learning (RL) promises to unlock the power of sequential decision-making by training agents entirely on static, pre-collected logs. This eliminates the need for expensive, risky, or unethical real-world exploration. However, while standard offline RL algorithms assume access to massive, highly diverse datasets, practical applications—ranging from clinical trials to robotics—frequently suffer from severe data scarcity.
When training data is limited, standard offline RL algorithms break down. Understanding why they fail, and how modern techniques overcome these limitations, requires looking closely at how agents evaluate actions they have never seen.
The Core Failure Mode: Value Hallucination
The fundamental challenge of offline RL is extrapolation error. When an agent updates its action-value function ($Q$-function) using the Bellman equation, it must evaluate the expected value of the next state by selecting the best possible action.
$$\max_{a} Q(s', a)$$
In a data-rich environment, the dataset contains enough samples to anchor the $Q$-function's predictions across most of the state-action space. But in a data-scarce environment, the dataset is full of gaps. If the agent queries a state-action pair $(s', a)$ that is missing from the dataset, the neural network approximating the $Q$-function has no data to constrain its prediction.
Because neural networks generalize imperfectly, these unconstrained regions often produce wildly inaccurate, overly optimistic values. The agent mistakes its own lack of knowledge for a high-reward opportunity. As Jeen et al. demonstrate in Zero-Shot Reinforcement Learning from Low Quality Data, this tendency to hallucinate high rewards for unseen actions is a direct symptom of data scarcity. Once the value function is corrupted by these hallucinations, the errors propagate backward through the Bellman updates, completely destabilizing the learning process.
The Limits of Pure Pessimism
To prevent value hallucination, traditional offline RL algorithms introduce pessimism. They penalize the value of actions that deviate from the behavior policy (the policy that generated the dataset).
While pessimism works well for large datasets, it paralyzes the agent when data is scarce. If a dataset contains only a few thousand trajectories, a strict pessimistic constraint forces the agent to rigidly mimic the data-generation policy. This defeats the primary purpose of offline RL: policy stitching. Policy stitching is the ability to take sub-optimal trajectories, extract the good segments from each, and combine them into a novel, optimal path.
As highlighted by Xu et al. in Sparse Q-Learning: Offline Reinforcement Learning with Implicit Value Regularization, environments with sparse, near-optimal trajectories require value functions that can stitch behaviors together without collapsing under out-of-distribution errors. When data is scarce, we cannot rely solely on penalizing the agent; we must find ways to fundamentally expand what the agent can safely understand.
Strategy 1: Generative Data Amplification
One way to overcome data scarcity is to fill the gaps in the dataset synthetically. Instead of relying on the agent to generalize during value estimation, we can train generative models to "upcycle" and expand the static dataset before or during training.
[Sparse Offline Dataset] ──> [Generative Model (Transformer/Diffusion)] ──> [Dense Synthetic Trajectories] ──> [Stable Value Learning]
Wang et al. introduced this concept via the Bootstrapped Transformer for Offline Reinforcement Learning (BooT). BooT leverages a learned sequence model to hallucinate its own high-quality training data. By generating trajectories, evaluating them, and filtering for high-confidence outcomes, the framework expands dataset coverage and allows sequence-based RL models to generalize far beyond the original limited logs.
Rather than just generating likely trajectories, we can also direct the generative process toward high-value regions. Huang et al. developed Goal-Conditioned Data Augmentation for Offline Reinforcement Learning (GODA), which uses diffusion models to synthesize transitions conditioned on high "return-to-go" targets. This effectively populates empty regions of the state-action space with highly optimized synthetic demonstrations, giving the offline agent a dense, high-quality landscape to learn from.
Strategy 2: Exploiting Fundamental Symmetries
Instead of generating entirely new data, another approach multiplies the value of existing data by embedding physical laws and structural constraints directly into the agent's architecture.
In many physical environments, transitions obey fundamental symmetries. For example, if an agent knows how to move forward from state $A$ to state $B$, the underlying physics of the environment often imply a corresponding reverse transition. Cheng et al. exploited this in Look Beneath the Surface: Exploiting Fundamental Symmetry for Sample-Efficient Offline RL by introducing Time-Reversal Symmetry (T-symmetry). By forcing the model to respect these bidirectional relationships, the authors showed that agents can learn highly effective policies using as little as 1% of standard training datasets.
The authors extended this insight to the problem of policy stitching in Sample Efficient Offline RL via T-Symmetry Enforced Latent State-Stitching (TELS). TELS maps sparse offline transitions into a compact, T-symmetry enforced latent space.
Standard Space: [State A] ──(Sparse/Disconnected)──> [State B] (Stitching Fails)
│
▼ (T-Symmetry Mapping)
Latent Space: [Latent A] <──(Symmetric Bridge)──> [Latent B] (Safe Stitching)
Because the latent space structurally enforces these geometric and physical symmetries, the agent can perform state-stitching smoothly within the latent representation. This bypasses the need for overly conservative action constraints and allows robust generalization even when the training set contains only a fraction of the typical transition volume.
Strategy 3: In-Sample Value Learning
A third strategy avoids value hallucination by changing how the Bellman optimality target is calculated. If the root cause of extrapolation error is querying the value of unseen actions ($\max_a Q(s', a)$), we can eliminate the error by avoiding the maximum operator entirely.
This technique, known as in-sample learning, constrains the evaluation of next-state values strictly to actions that actually appear within the dataset. Xu et al. implemented this implicitly in Sparse Q-Learning through an Implicit Value Regularization (IVR) framework.
Instead of explicitly modeling a policy constraint or penalizing out-of-distribution actions, Sparse Q-Learning introduces sparsity directly into the value learning objective. This mathematical formulation allows the value function to upper-bound the data distribution's returns using only the state-action samples present in the log. Because the algorithm never evaluates hypothetical actions, it remains completely immune to value hallucination, making it highly effective at extracting policies from sparse, low-density datasets.
Key Foundational Work
Model-Free Least-Squares Policy Iteration (2001) — contributed 1 concept elements to this direction.
Learning to Teach with a Reinforcement Learning Agent (1998) — contributed 1 concept elements to this direction.