Offline Reinforcement Learning under Partial Data Coverage

This direction is merged into another direction: concentrability assumption. Its last version stays readable and citable here.

Version 1 of 138 papers40 conceptsfrozen 2026-10-04

In traditional Reinforcement Learning (RL), an agent learns by trial and error. If it wonders whether a specific action is good, it simply tries it. In Offline RL, this luxury disappears. The agent is given a fixed dataset—often collected by a human or a legacy system—and must derive an optimal strategy without ever taking a new exploratory step.

The fundamental theoretical hurdle in this field has long been the coverage assumption. Early proofs required "uniform concentrability," a demanding condition where the dataset must explore almost every possible state and action. If the data is "narrow"—for instance, if a self-driving car dataset only contains safe highway driving—traditional algorithms might "hallucinate" that driving off a cliff is a high-reward shortcut simply because the dataset contains no evidence to the contrary.

Recent research has shifted toward a more realistic paradigm: learning effectively when the data only provides partial coverage.

The Principle of Pessimism

The core mechanism for handling partial coverage is pessimism. When an agent encounters a state-action pair it hasn't seen before (or has seen rarely), it should not assume the average reward; instead, it should assume the worst-case scenario.

As Masatoshi Uehara and Wen Sun demonstrate in Pessimistic Model-based Offline RL, pessimism acts as a regularizer that penalizes the agent for straying into "epistemic uncertainty"—regions where the data is too thin to make reliable predictions. By constructing lower confidence bounds on the value function, the agent is incentivized to stay within the "safe" support of the offline data.

This principle is not limited to standard RL. In PASTA: Pessimistic Assortment Optimization, Juncheng Dong and colleagues apply this to product recommendation, showing that pessimism prevents overestimating the revenue of untested product combinations.

From Global to Single-Policy Concentrability

The most significant theoretical breakthrough in this area is the relaxation of "all-policy" coverage to single-policy coverage.

Under the old "all-policy" requirement, the dataset had to cover the state-action distributions of every possible policy. This is practically impossible in high-dimensional spaces. The new frontier, explored by researchers like Hanlin Zhu and Amy Zhang in Provably Efficient Offline Goal-Conditioned RL, only requires that the dataset covers the actions of a single good policy (like the expert's).

If the data contains at least one path to the goal, these new algorithms can find it, even if 99% of the state space remains a mystery. This shift makes offline learning viable for complex tasks like robotic control or healthcare, where we only have logs of "good" behavior rather than exhaustive exploration.

Algorithmic Paths to Robustness

Researchers have developed several distinct mathematical frameworks to implement this "partial coverage" logic:

  1. Minimax Optimization: Instead of simple regression, algorithms like MSQP and MQP frame Q-learning as a game against an adversary. As shown in Refined Value-Based Offline RL by Nathan Kallus and Jason D. Lee, this approach provides $L^2$-convergence guarantees without needing the restrictive "Bellman completeness" assumption.
  2. Linear Programming with Constraints: By revisiting the classic LP formulation of MDPs, Sarath Pattathil and Kaiqing Zhang introduced error-bound induced constraints. In Offline RL via Linear-Programming with Error-Bound Induced Constraints, they show this achieves optimal sample complexity while naturally handling partial data.
  3. Distributionally Robust Learning (DRL): This approach prepares the agent for "distribution shift"—the inevitable difference between the training data and the deployment environment. Kishan Panaganti and Mohammad Ghavamzadeh demonstrate in Bridging Distributionally Robust Learning and Offline RL that DRL can mathematically guarantee performance even when the behavior policy that collected the data was highly biased.

Shattering Impossibility Results

For years, it was conjectured that learning an optimal policy with only "realizability" (the assumption that your function class contains the true value function) was information-theoretically impossible without full coverage.

Tengyang Xie and Nan Jiang shattered this notion in Batch Value-function Approximation with Only Realizability. By introducing the Batch Value-Function Tournament (BVFT), they proved that polynomial sample efficiency is possible under minimal assumptions. This work, along with the development of Pessimistic Q-Learning by Gen Li and Yuting Wei, has closed the gap between theoretical "worst-case" pessimism and practical "best-case" performance.

Papers in this version

History & citationVersion 1 · 2026-10-04

Merged into another direction · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_offline_reinforcement_learning_under_par_756c099f,
  title = {Offline Reinforcement Learning under Partial Data Coverage},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/offline-reinforcement-learning-under-partial-data-coverage/rdr_756c099fc1ba4e168d4d68cead9a0c75},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_offline_reinforcement_learning_under_par_756c099f_v1,
  title = {Offline Reinforcement Learning under Partial Data Coverage},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/offline-reinforcement-learning-under-partial-data-coverage/rdr_756c099fc1ba4e168d4d68cead9a0c75/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}