In standard online reinforcement learning (RL), an agent learns by trial and error. If it incorrectly estimates that an unfamiliar action will yield a massive reward, it attempts that action, receives immediate feedback from the environment, and corrects its mistake.
In offline reinforcement learning, however, this corrective feedback loop is entirely broken. The agent must learn an optimal policy exclusively from a static, pre-collected dataset. If the agent's neural network value function (the Q-network) erroneously predicts a high reward for an action not present in the training data, the agent cannot execute the action to verify it. Instead, the learning algorithm blindly accepts this hallucinated value, leading to a phenomenon known as extrapolation error or out-of-distribution (OOD) value overestimation.
When deployed in the real world, policies corrupted by these inflated expectations fail catastrophically because they rely on transitions and rewards the agent has never actually encountered.
The Mechanics of Extrapolation Error
To understand why value functions systematically overestimate unseen actions, we have to look at the core mathematical engine of Q-learning: the Bellman optimality backup. During training, the Q-function is updated iteratively using the target:
$$Y = r + \gamma \max_{a'} Q(s', a')$$
The presence of the $\max$ operator introduces a severe selection bias when paired with neural network function approximation. Consider an action $a'$ that lies completely outside the distribution of the training dataset. Because the neural network has never received gradient updates for this state-action pair, its prediction is ungrounded—essentially a random guess determined by the network's initialization and inductive biases.
If the network happens to underestimate the value of this OOD action, the $\max$ operator will simply ignore it. However, if the network happens to overestimate its value, the $\max$ operator will greedily select it as the target for the update. As highlighted by Aviral Kumar et al. in Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reduction, this overestimation error is then bootstrapped back into the values of preceding states. Over thousands of training iterations, these positive errors accumulate and propagate backward through the state-action trajectory, causing the value function to completely diverge.
To solve this problem, the offline RL community has converged on two primary paradigms: explicit pessimism (fixing the value function) and policy constraints (fixing the action selection).
Paradigm 1: Explicit Pessimism and Value Regularization
If the root cause of policy failure is that the Q-function hallucinates high rewards in unrepresented regions, a direct solution is to enforce pessimism. By intentionally penalizing the value function in areas of low data coverage, we can ensure that the $\max$ operator never chooses an OOD action.
As demonstrated by Gen Li et al. in Pessimistic Q-Learning for Offline Reinforcement Learning: Towards Optimal Sample Complexity, incorporating a pessimistic penalty directly into model-free Q-learning allows the algorithm to achieve minimax-optimal sample complexity, proving that conservative value estimation is theoretically sufficient to overcome data limitations.
Researchers have implemented this value-level pessimism in several ways:
- State-Value Penalization: Instead of only penalizing individual state-action pairs, Liting Chen et al. developed Conservative State Value Estimation for Offline Reinforcement Learning (CSVE). This approach uses a dynamics model to sample unfamiliar states and directly penalizes the state value $V$, providing tighter theoretical bounds and preventing the agent from steering toward dangerous, unrepresented regions of the state space.
- Uncertainty-Aware Weighting: In environments with massive action spaces, such as recommendation engines, Chengqian Gao et al. proposed Value Penalized Q-Learning for Recommender Systems (VPQ). VPQ quantifies the uncertainty of Q-value predictions and applies a proportional penalty, effectively neutralizing unstable estimates without needing to explicitly model complex behavior policies.
Paradigm 2: Policy Constraints and In-Sample Learning
An alternative strategy is to leave the Q-function unpenalized but restrict the policy's freedom. If the policy is strictly forbidden from choosing actions outside the dataset's support, the agent will never encounter the ungrounded regions where extrapolation error occurs.
Early methods achieved this by adding an explicit distance penalty (such as KL-divergence) between the learned policy and an estimated behavior policy. However, modeling the behavior policy accurately can be incredibly difficult in complex or multimodal datasets.
To bypass this bottleneck, recent work focuses on in-sample learning, where the algorithm evaluates the value function using only the actions that actually appear in the dataset.
- The In-Sample Softmax: Chenjun Xiao et al. introduced this concept in The In-Sample Softmax for Offline Reinforcement Learning. By replacing the standard $\max$ or expectile operators with a softmax calculated strictly over the actions present in the training logs, their algorithm (InAC) completely eliminates the need to query OOD actions during the Bellman backup.
- Implicit Value Updates: Similarly, Zifan LIU et al. utilized implicit value updates in C2IQL: Constraint-Conditioned Implicit Q-learning for Safe Offline Reinforcement Learning. By avoiding explicit maximization over the action space entirely, they prevent the policy from exploiting hallucinated rewards, enabling safe policy optimization under strict real-world constraints.
Bypassing Algorithmic Complexity
While explicit pessimism and policy constraints modify the core RL loss functions, a parallel line of research suggests that extrapolation error can sometimes be mitigated by changing the scale of optimization rather than the algorithm itself.
For instance, Alexander Nikulin et al. discovered in Q-Ensemble for Offline RL: Don't Scale the Ensemble, Scale the Batch Size that the community's reliance on massive ensembles of hundreds of Q-functions to enforce conservatism is often unnecessary. They demonstrated that simply scaling the mini-batch size to 10,000 provides a natural regularizing effect that keeps value estimates grounded while significantly reducing training time.
From a architectural perspective, Edoardo Cetin demonstrated in Simple Ingredients for Offline Reinforcement Learning that standard offline algorithms often fail on large, heterogeneous datasets because of capacity limitations. By simply scaling up the parameter size of the policy network, basic algorithms can generalize across diverse data distributions without succumbing to the extrapolation errors that typically cause policy collapse.