Mitigating Extrapolation Error in Offline Reinforcement Learning

You are reading version 1 of this direction, frozen 2026-10-04. The latest version is version 2.

Version 1 of 2517 papers610 conceptsfrozen 2026-10-04

In standard reinforcement learning (RL), an agent learns by doing. It explores an environment, receives rewards, and corrects its mistakes through trial and error. However, in many real-world applications—like healthcare, autonomous driving, or industrial robotics—this kind of "online" exploration is too dangerous, expensive, or unethical.

Offline Reinforcement Learning (also known as Batch RL) attempts to learn an optimal policy from a fixed, pre-collected dataset of transitions, without any further interaction with the environment. While this sounds like a standard supervised learning problem, it introduces a fundamental failure mode: extrapolation error.

The Hallucination of High Rewards

The core of the problem lies in the way RL algorithms evaluate actions. Most algorithms use a Q-function to estimate the "value" (expected future reward) of a state-action pair. During training, the agent tries to find the action that maximizes this value.

In an offline setting, the agent is presented with a static dataset. When the agent's policy considers an action that is not well-represented in that dataset—an out-of-distribution (OOD) action—the neural network representing the Q-function has no data to ground its prediction. Because of the way neural networks generalize, they often assign erroneously high values to these unseen actions.

As Scott Fujimoto and Edoardo Conti identify in Benchmarking Batch Deep Reinforcement Learning Algorithms, this extrapolation error creates a feedback loop. The agent "hallucinates" that an unknown action is brilliant, selects it, and then uses that inflated value to update other states via bootstrapping. This leads to divergent value estimates and a total collapse in performance when the agent is actually deployed.

Policy Constraints: Staying Within the Lines

The most intuitive way to fix this is to prevent the agent from ever straying too far from the actions seen in the dataset. This is known as policy regularization or policy constraints.

A "minimalist" version of this idea was proposed by Scott Fujimoto and Shixiang Shane Gu in A Minimalist Approach to Offline Reinforcement Learning. They demonstrated that simply adding a Behavior Cloning (BC) term to the policy update—forcing the agent to stay close to the data—allows standard algorithms like TD3 to match state-of-the-art offline performance.

Other researchers have refined this by focusing on the "support" of the data rather than its density. In Supported Trust Region Optimization for Offline Reinforcement Learning, Yixiu Mao and colleagues argue that we shouldn't just mimic the behavior policy; we should be free to choose any action as long as it exists within the set of actions the dataset "supports."

Conservative Value Estimation: The Pessimistic Critic

Instead of constraining the actor (the policy), another school of thought constrains the critic (the value function). This is often called pessimism. If the agent encounters an action it hasn't seen before, the critic should assume the worst rather than the best.

Ilya Kostrikov and others explored this in Offline Reinforcement Learning with Fisher Divergence Critic Regularization, where they regularize the critic's gradients to prevent over-optimism. Similarly, Yixiu Mao proposed Supported Value Regularization (SVR), which mathematically penalizes OOD actions while leaving in-distribution actions untouched, providing a theoretical guarantee of safety.

The tension between these two approaches—constraining the policy versus regularizing the critic—is a central theme in the field. Benjamin Eysenbach and Sergey Levine have even shown a formal connection between one-step policy regularization and critic regularization, suggesting they are two sides of the same coin.

Trajectory Stitching: Making the Most of Suboptimal Data

One of the most powerful "superpowers" of offline RL is trajectory stitching. Imagine a dataset where one driver knows how to get from point A to B, and another driver knows how to get from B to C. Neither driver has ever gone from A to C.

A simple imitation learning model (Behavior Cloning) would only learn to replicate the individual paths. However, an offline RL agent can "stitch" these segments together. By correctly estimating the value of being at point B, the agent can realize that the first driver's path leads to a state where the second driver's path becomes available.

This concept is central to Model-based trajectory stitching for improved behavioural cloning and its applications by Charles Alexander Hepburn. By using models to bridge gaps in the data, agents can discover optimal sequences of behavior that were never actually performed by the original data collectors.

Why It Works: The Balance of Bias and Variance

Ultimately, all these methods are trying to manage the trade-off between exploitation (finding the best action in the data) and caution (avoiding the unknown).

Algorithms like Quantile Filtered Imitation Learning (QFIL) by David Brandfonbrener use hard filters to only imitate high-value actions, while Implicit Q-Learning (IQL) avoids querying OOD actions entirely during training. Whether through policy constraints, pessimistic critics, or data augmentation, the goal is the same: to ground the agent's ambitions in the reality of the evidence provided.

Papers in this version

Showing 60 of 517 papers.

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 370 papers
  2. Version 1 · 2026-10-04 · 517 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_extrapolation_error_out_of_distribution__f6ab0eae,
  title = {extrapolation error (out-of-distribution value overestimation)},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/extrapolation-error-out-of-distribution-value-overestimation/rdr_f6ab0eae53734743946d09ca24b000ca},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 1, frozen text):

@misc{lacuna_mitigating_extrapolation_error_in_offlin_f6ab0eae_v1,
  title = {Mitigating Extrapolation Error in Offline Reinforcement Learning},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/mitigating-extrapolation-error-in-offline-reinforcement-learning/rdr_f6ab0eae53734743946d09ca24b000ca/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-21},
}