Offline Reinforcement Learning from Fixed Interaction Logs

25 concepts24 papers1 proposal

Standard reinforcement learning (RL) is fundamentally active: an agent interacts with an environment, observes a reward, and continuously updates its strategy via trial and error. But in many real-world settings, active exploration is impossible. A bad exploratory action could trigger an epileptic seizure, crash a cellular network, lose substantial capital in financial markets, or alien to a student using an educational platform.

In these safety-critical and high-cost domains, we cannot afford live exploration. However, we often possess vast repositories of historical data—logs of how a baseline policy previously managed the system. As argued by Levine in Understanding the World Through Action, shifting from active trial-and-error to learning control policies entirely from these pre-existing datasets allows RL to leverage the scaling benefits of big data while bypassing physical and ethical constraints. This paradigm is known as offline (or batch) reinforcement learning.

Extracting Value via Dynamic Programming

How do we learn to make optimal decisions when we cannot gather new data? The core mechanism relies on performing dynamic programming over a fixed buffer of transitions $\mathcal{D} = \{(s, a, r, s')\}$.

Instead of stepping through an environment in real time, algorithms like Fitted Q-Iteration (FQI) convert value estimation into a sequence of supervised regression problems. In each iteration, the target value for a state-action pair is computed using the recorded reward $r$ and the discounted maximum expected value of the next state $s'$. To generalize across continuous or high-dimensional state spaces, this value function is paired with robust function approximators.

For instance, Guez et al. in Adaptive Treatment of Epilepsy via Batch-mode Reinforcement Learning and Bisi et al. in Foreign exchange trading: a risk-averse batch reinforcement learning approach utilize Extremely Randomized Trees to stably map complex states to actions without real-time interaction. In recommendation systems, Theocharous et al. leverage random forests in Personalized Ad Recommendation Systems for Life-Time Value Optimization with Guarantees to optimize long-term user life-time value from static historical data.

This strategy is equally effective for physical infrastructure. Castiglione et al. apply multi-step FQI in Reinforcement Learning Control Algorithm for HVAC Retrofitting: Application to a Supermarket Building Model by Dynamic Simulation, and Wen et al. use deep neural networks in Transforming Cooling Optimization for Green Data Center via Deep Reinforcement Learning to safely minimize cooling costs. Similarly, Berlink et al. demonstrate in Batch Reinforcement Learning for Smart Home Energy Management that agents can learn to manage surplus solar energy safely by training exclusively on historical consumption and price trends.

The Challenge of Distribution Shift

If we simply maximize the estimated Q-value over a fixed dataset, standard RL algorithms fail catastrophically due to distribution shift. When evaluating a new policy, the Bellman update queries the value of the next state-action pair: $\max_{a'} Q(s', a')$. If an action $a'$ was never taken in the historical logs, the function approximator has to extrapolate blindly. This often results in wildly inaccurate, overly optimistic value hallucinations, causing the agent to select unvalidated and potentially dangerous actions.

To counter this, offline RL must constrain the policy to stay within the support of the historical data distribution. As Zou et al. note in RL4RS: A Real-World Dataset for Reinforcement Learning based Recommender System, constraining the policy to the support of the training data mitigates the risk of overestimating unseen actions. This constraint is vital when refining production recommendation engines, as highlighted by Ji et al. in Correction to: Learning to bid and rank together in recommendation systems, where offline RL refines joint bidding and ranking policies using historical interaction logs without live production risks.

Standard RL:      [State s] ---> (Query Max Action a') ---> [Hallucinated High Q-Value] ❌

Offline Constrained: [State s] ---> (Restrict to Data Support) ---> [Safe, Validated Action]        

In user-facing applications like language modeling, this alignment is often framed as counterfactual optimization. User interactions (such as accepting or ignoring a suggestion) provide sparse, binary rewards. Arnold et al. in Counterfactual Language Model Adaptation for Suggesting Phrases and Lawrence et al. in Improving a Neural Semantic Parser by Counterfactual Learning from Human Bandit Feedback show that models can be optimized off-policy by applying counterfactual risk minimization to logged feedback. This mathematically corrects for the difference between the historical behavior policy and the new target policy, bypassing live human testing.

For large-scale dialogue agents, Shea et al. introduce importance sampling techniques (such as VaRMI) in Building Persona Consistent Dialogue Agents with Offline Reinforcement Learning to enforce persona consistency, avoiding the high generation costs and instability of online training loops.

Overcoming Computational Bottlenecks

Decoupling policy learning from data collection introduces unique structural challenges. Because the dataset is static and often massive, the computational complexity of updating the value function can scale poorly with the number of features.

To address this, researchers use structural preprocessing to isolate feature selection from policy iteration. In spoken dialogue systems, Chandramohan et al. introduce a static dictionary construction technique in Sparse Approximate Dynamic Programming for Dialog Management. By identifying the most representative data points to act as precomputed basis functions before policy iteration begins, they reduce the computational complexity of value function updates from cubic to quadratic.

This decoupling allows algorithms like Least Squares Policy Iteration (LSPI) to run efficiently on fixed logs. For example, Daubigney et al. demonstrate in Optimisation d'un tuteur intelligent à partir d'un jeu de données fixé (Optimization of a tutoring system from a fixed set of data) [in French] that tutoring systems can derive optimal testing versus teaching strategies directly from fixed logs, avoiding the need for complex, hand-coded cognitive models.

Furthermore, policy gradient methods can be adapted to the offline setting. As shown by Kandasamy et al. in Batch Policy Gradient Methods for Improving Neural Conversation Models, batch policy gradient methods effectively utilize pre-recorded reward trajectories to optimize language generation models stably, outperforming online variants when restricted to static data.

This stability enables offline optimization to scale to complex human-agent collaboration tasks. For instance, the ReHAC framework developed by Feng et al. in Large Language Model-based Human-Agent Collaboration for Complex Task Solving trains LLM agents to decide exactly when to delegate tasks to human supervisors. By training entirely on pre-recorded interaction logs, the framework optimizes complex collaboration policies without incurring the temporal and financial costs of active, human-in-the-loop training loops.

Key Foundational Work

Optimizing spoken dialogue management with fitted value iteration (2010) — contributed 1 concept elements to this direction.
Adaptive Treatment of Epilepsy via Batch-mode Reinforcement Learning (2008) — contributed 1 concept elements to this direction.
Sparse Approximate Dynamic Programming for Dialog Management (2010) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_batch_reinforcement_learning_a2ebefed,
  title = {batch reinforcement learning},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/batch-reinforcement-learning/rdr_a2ebefedfde648a88c43433a92b2eb7c},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_batch_reinforcement_learning_a2ebefed_v1,
  title = {batch reinforcement learning},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/batch-reinforcement-learning/rdr_a2ebefedfde648a88c43433a92b2eb7c/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake