Distribution Shift in Offline Reinforcement Learning

1,334 concepts944 papers1 proposal

In standard reinforcement learning (RL), an agent learns by trial and error, interacting with an environment to discover which actions yield the highest rewards. However, in many real-world applications—such as clinical medicine or power system dispatch—active exploration is either too dangerous or too expensive. Offline RL seeks to learn optimal policies entirely from static, pre-collected datasets.

The fundamental obstacle in this setting is distribution shift. When an agent attempts to improve upon the behavior policy that collected the data, it inevitably considers actions that are not well-represented in the training set. Because the agent cannot interact with the environment to verify the outcomes of these "out-of-distribution" (OOD) actions, it often falls into a trap of catastrophic over-optimism.

The Mechanism of Value Overestimation

The core of the problem lies in how value-based RL algorithms, like Q-learning, update their estimates. These algorithms use a "max" operator to select the best possible future action when calculating the value of a current state. In an offline setting, if the function approximator (such as a neural network) makes a slight upward error in estimating the value of an OOD action, the max operator will aggressively select that action.

As Kumar et al. note in their work on the D4RL benchmark, this leads to a "delusional" feedback loop. The agent begins to believe that unfamiliar regions of the state-action space are highly rewarding simply because it lacks the data to prove otherwise. Without corrective feedback from the environment, these errors accumulate, causing the learned policy to collapse during deployment as the agent pursues hallucinated rewards.

Action vs. State Distribution Shift

Distribution shift occurs at two levels:

  1. Action Shift: The learned policy selects an action $a$ in a known state $s$ that the behavior policy rarely or never performed. Kumar et al. proposed the BEAR algorithm to address this by constraining the agent to only choose actions within the "support" of the training data.
  2. State Shift: Even if actions are constrained, small errors in the policy can lead the agent into entirely unfamiliar states. As the authors of BEAR point out, simply staying within the action support does not guarantee the agent won't drift into OOD states where its value estimates are unreliable. Chen et al. addressed this by introducing Conservative State Value Estimation (CSVE), which directly penalizes the value of unfamiliar states sampled via a dynamics model.

Inducing Pessimism

To counter over-optimism, the field has converged on the principle of pessimism. The goal is to ensure that the agent's value estimates are a lower bound on the true values for OOD actions.

  • Conservative Q-Learning (CQL): This approach, utilized in systems like Q-Transformer, regularizes the Q-function to stay low on OOD actions while maintaining high values for actions present in the dataset.
  • Bellman-consistent Pessimism: Xie et al. argue that traditional methods can be "over-pessimistic," hindering performance. They propose enforcing pessimism only on value functions that remain consistent with the Bellman equations, balancing data coverage with policy improvement.
  • Model-Based Conservatism: In model-based offline RL, the agent learns a "world model" from the data. Yu et al. introduced COMBO, which regularizes the value function using synthetic data generated by the model, penalizing transitions where the model is uncertain.

Beyond Value Penalties

Alternative frameworks avoid explicit Q-value penalties by reframing the problem. Lee et al. developed OptiDICE, which bypasses the unstable Bellman update entirely. Instead, it directly estimates the stationary distribution of the optimal policy using stationary distribution correction, transforming the RL problem into a stable convex optimization.

Similarly, Xu et al. propose IDRL, which treats offline RL as a weighted imitation learning problem. By using an "optimal discriminator" to filter out suboptimal transitions in the static dataset, the authors achieve stability without the need for complex conservative penalties.

The Importance of the Discount Factor

Interestingly, the choice of hyperparameters can also serve as a defense against distribution shift. Hu et al. demonstrated that the discount factor ($\gamma$), usually a fixed value, acts as a regularizer in offline settings. A smaller $\gamma$ reduces the agent's reliance on long-term future estimates, which are more likely to be corrupted by accumulated OOD errors, thereby inducing a form of implicit conservatism.

History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_distribution_shift_d81b3269,
  title = {distribution shift},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/distribution-shift/rdr_d81b3269f5f94dc4880707d8b75276b4},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_distribution_shift_d81b3269_v1,
  title = {distribution shift},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/distribution-shift/rdr_d81b3269f5f94dc4880707d8b75276b4/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake