Out of Distribution Overestimation in Offline Reinforcement Learning

521 concepts343 papers4 proposals

In standard online reinforcement learning (RL), an agent learns by trial and error. If it incorrectly estimates that an unfamiliar action will yield a massive reward, it attempts that action, receives immediate feedback from the environment, and corrects its mistake.

In offline reinforcement learning, however, this corrective feedback loop is entirely broken. The agent must learn an optimal policy exclusively from a static, pre-collected dataset. If the agent's neural network value function (the Q-network) erroneously predicts a high reward for an action not present in the training data, the agent cannot execute the action to verify it. Instead, the learning algorithm blindly accepts this hallucinated value, leading to a phenomenon known as extrapolation error or out-of-distribution (OOD) value overestimation.

When deployed in the real world, policies corrupted by these inflated expectations fail catastrophically because they rely on transitions and rewards the agent has never actually encountered.

The Mechanics of Extrapolation Error

To understand why value functions systematically overestimate unseen actions, we have to look at the core mathematical engine of Q-learning: the Bellman optimality backup. During training, the Q-function is updated iteratively using the target:

$$Y = r + \gamma \max_{a'} Q(s', a')$$

The presence of the $\max$ operator introduces a severe selection bias when paired with neural network function approximation. Consider an action $a'$ that lies completely outside the distribution of the training dataset. Because the neural network has never received gradient updates for this state-action pair, its prediction is ungrounded—essentially a random guess determined by the network's initialization and inductive biases.

If the network happens to underestimate the value of this OOD action, the $\max$ operator will simply ignore it. However, if the network happens to overestimate its value, the $\max$ operator will greedily select it as the target for the update. As highlighted by Aviral Kumar et al. in Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reduction, this overestimation error is then bootstrapped back into the values of preceding states. Over thousands of training iterations, these positive errors accumulate and propagate backward through the state-action trajectory, causing the value function to completely diverge.

To solve this problem, the offline RL community has converged on two primary paradigms: explicit pessimism (fixing the value function) and policy constraints (fixing the action selection).

Paradigm 1: Explicit Pessimism and Value Regularization

If the root cause of policy failure is that the Q-function hallucinates high rewards in unrepresented regions, a direct solution is to enforce pessimism. By intentionally penalizing the value function in areas of low data coverage, we can ensure that the $\max$ operator never chooses an OOD action.

As demonstrated by Gen Li et al. in Pessimistic Q-Learning for Offline Reinforcement Learning: Towards Optimal Sample Complexity, incorporating a pessimistic penalty directly into model-free Q-learning allows the algorithm to achieve minimax-optimal sample complexity, proving that conservative value estimation is theoretically sufficient to overcome data limitations.

Researchers have implemented this value-level pessimism in several ways:

  • State-Value Penalization: Instead of only penalizing individual state-action pairs, Liting Chen et al. developed Conservative State Value Estimation for Offline Reinforcement Learning (CSVE). This approach uses a dynamics model to sample unfamiliar states and directly penalizes the state value $V$, providing tighter theoretical bounds and preventing the agent from steering toward dangerous, unrepresented regions of the state space.
  • Uncertainty-Aware Weighting: In environments with massive action spaces, such as recommendation engines, Chengqian Gao et al. proposed Value Penalized Q-Learning for Recommender Systems (VPQ). VPQ quantifies the uncertainty of Q-value predictions and applies a proportional penalty, effectively neutralizing unstable estimates without needing to explicitly model complex behavior policies.

Paradigm 2: Policy Constraints and In-Sample Learning

An alternative strategy is to leave the Q-function unpenalized but restrict the policy's freedom. If the policy is strictly forbidden from choosing actions outside the dataset's support, the agent will never encounter the ungrounded regions where extrapolation error occurs.

Early methods achieved this by adding an explicit distance penalty (such as KL-divergence) between the learned policy and an estimated behavior policy. However, modeling the behavior policy accurately can be incredibly difficult in complex or multimodal datasets.

To bypass this bottleneck, recent work focuses on in-sample learning, where the algorithm evaluates the value function using only the actions that actually appear in the dataset.

Bypassing Algorithmic Complexity

While explicit pessimism and policy constraints modify the core RL loss functions, a parallel line of research suggests that extrapolation error can sometimes be mitigated by changing the scale of optimization rather than the algorithm itself.

For instance, Alexander Nikulin et al. discovered in Q-Ensemble for Offline RL: Don't Scale the Ensemble, Scale the Batch Size that the community's reliance on massive ensembles of hundreds of Q-functions to enforce conservatism is often unnecessary. They demonstrated that simply scaling the mini-batch size to 10,000 provides a natural regularizing effect that keeps value estimates grounded while significantly reducing training time.

From a architectural perspective, Edoardo Cetin demonstrated in Simple Ingredients for Offline Reinforcement Learning that standard offline algorithms often fail on large, heterogeneous datasets because of capacity limitations. By simply scaling up the parameter size of the policy network, basic algorithms can generalize across diverse data distributions without succumbing to the extrapolation errors that typically cause policy collapse.

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 370 papers
  2. Version 1 · 2026-10-04 · 517 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_extrapolation_error_out_of_distribution__f6ab0eae,
  title = {extrapolation error (out-of-distribution value overestimation)},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/extrapolation-error-out-of-distribution-value-overestimation/rdr_f6ab0eae53734743946d09ca24b000ca},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 2, frozen text):

@misc{lacuna_extrapolation_error_out_of_distribution__f6ab0eae_v2,
  title = {extrapolation error (out-of-distribution value overestimation)},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/extrapolation-error-out-of-distribution-value-overestimation/rdr_f6ab0eae53734743946d09ca24b000ca/v/2},
  note = {Version 2, frozen 2026-10-04. Accessed 2026-11-21},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake