In standard online reinforcement learning (RL), an agent learns by trial and error. If it overestimates the value of a bad action, it takes that action, receives a poor reward, and corrects its internal value function (the Q-function). However, in offline reinforcement learning, the agent is completely insulated from the environment; it must learn exclusively from a static, pre-collected dataset.
This lack of interaction exposes a fundamental vulnerability in the standard Bellman optimality equation:
$$Q(s, a) = r + \gamma \max_{a'} Q(s', a')$$
When updating the value of a state-action pair, the $\max_{a'}$ operator queries the Q-function for the best possible next action. If the policy selects an out-of-distribution (OOD) action—an action underrepresented or entirely absent from the training data—the Q-function has no empirical data to anchor its prediction. Because deep neural networks generalize poorly to OOD inputs, they frequently "hallucinate" high rewards for these unfamiliar actions. The max operator eagerly propagates these overestimations through the Bellman backups, causing the policy to exploit its own hallucinations and fail catastrophically upon deployment.
To solve this, a prominent research direction focuses on grounding the agent's learning process within the empirical support of the dataset. This is achieved through three primary mechanisms: penalizing the value function, regularizing the policy space, or redesigning the mathematical operators to avoid OOD queries entirely.
Conservative Value Estimation: Lower-Bounding the Q-Function
The most direct way to combat hallucinated rewards is to explicitly penalize the Q-values of OOD actions during training, forcing the learned value function to act as a safe lower bound.
For example, Zhihao Cheng et al. incorporate a conservative penalty term into the loss function of quantum agents in Offline Quantum Reinforcement Learning in a Conservative Manner to suppress the estimated values of actions not present in the dataset. This mechanism ensures that the agent remains pessimistic about what it has not seen.
This principle of value pessimism is vital across high-stakes, real-world domains where unverified actions can be dangerous:
- Healthcare: In optimizing sedative dosing for ICU patients, Hong Yeul Lee et al. demonstrate in Reinforcement learning model for optimizing dexmedetomidine dosing to prevent delirium in critically ill patients that penalizing unobserved actions prevents the model from recommending potentially unsafe clinical interventions.
- Telecommunications: Kun Yang et al. apply pessimistic value estimation in Advancing RAN Slicing with Offline Reinforcement Learning to avoid overoptimistic resource allocation strategies in cellular networks.
- Automated Negotiation: Siqi Chen et al. show in An effective negotiating agent framework based on deep offline reinforcement learning that conservative value estimation preserves stable negotiation strategies across diverse domains without requiring risky live market interactions.
This penalization strategy also scales effectively to distributed and imitation learning settings. In federated learning, where data is siloed across multiple clients, Sheng Yue et al. introduce a dual regularization framework in Federated Offline Policy Optimization with Dual Regularization to penalize OOD actions during local updates. To handle distributed datasets of mixed quality, Nan Qiao et al. utilize a similar conservative value framework in FOVA: Offline Federated Reinforcement Learning with Mixed-Quality Data to mitigate overoptimism. Even when learning from highly suboptimal demonstrations, Huy Hoang et al. show in SPRINQL: Sub-optimal Demonstrations driven Offline Imitation Learning and SubIQ: Inverse Soft-Q Learning for Offline Imitation with Suboptimal Demonstrations that integrating a conservative term into the value function prevents the policy from drifting into unsupportable states where performance degrades unpredictably.
Policy Constraints: Staying Close to the Data
Instead of modifying the value function, a complementary strategy restricts the policy directly, ensuring the actor never selects actions far from the behavior policy that generated the dataset.
A minimalist yet highly effective way to achieve this is by adding a behavioral cloning (BC) loss directly to the actor's objective. As explored by Alex Beeson et al. in Improving TD3-BC: Relaxed Policy Constraint for Offline Learning and Stable Online Fine-Tuning, this simple regularization penalizes deviations from the training dataset without requiring complex density estimation or importance sampling. The authors demonstrate that strategically relaxing this constraint after initial training allows for stable online fine-tuning.
This minimalist regularization is highly versatile: Cong Lu et al. show in Challenges and Opportunities in Offline Reinforcement Learning from Visual Observations that a simple BC regularizer successfully mitigates catastrophic overestimation in vision-based offline RL. Furthermore, when dealing with complex, multi-dimensional action spaces, Alex Beeson et al. demonstrate in An Investigation of Offline Reinforcement Learning in Factorisable Action Spaces that applying these constraints at the sub-action level effectively manages OOD risks without suffering from a combinatorial explosion.
For environments where a simple BC penalty is too restrictive, researchers use more sophisticated data-anchoring mechanisms:
- Digital Twins: In Digital Twin-Enhanced Deep Reinforcement Learning for Resource Management in Networks Slicing, Yongming Huang et al. employ a regularization penalty within a high-fidelity virtual sandbox to keep the policy safe before live deployment.
- Generative Anchoring: Generative models can be trained to model the behavior policy explicitly. Yang Zhang et al. in BCORLE(λ): An Offline Reinforcement Learning and Evaluation Framework for Coupons Allocation in E-commerce Market and Núria Armengol Urpí et al. in Risk-Averse Offline Reinforcement Learning use generative models to restrict action selection to small perturbations of observed behaviors, filtering out actions that the model might otherwise incorrectly estimate as high-reward.
- Pseudometric Learning: Rather than generative modeling, Robert Dadashi et al. propose a metric approach in Offline Reinforcement Learning with Pseudometric Learning, learning a task-aware distance measure to penalize actions based on their distance to the nearest neighbor in the logged transitions.
In-Sample Learning: Eliminating OOD Queries Entirely
While value penalization and policy constraints mitigate the symptoms of distribution shift, in-sample learning eliminates the root cause by redesigning the learning operators to avoid querying OOD actions altogether.
Consider the standard Bellman backup, which evaluates $\max_{a'} Q(s', a')$. In-sample learning replaces this unconstrained maximization with operators that evaluate values only on actions present within the dataset. As developed by Chenjun Xiao et al. in The In-Sample Softmax for Offline Reinforcement Learning, the In-Sample Softmax operator applies a log-sum-exp function exclusively over the empirical support of the dataset. This mathematically guarantees that the value function never encounters an OOD action query during training.
Standard Bellman Backup:
Query Q(s', a') ---> Can choose ANY action ---> High risk of OOD Overestimation
In-Sample Learning:
Query Q(s', a') ---> Restricted to actions in dataset ---> Zero risk of OOD Overestimation
Providing a theoretical foundation for this paradigm, Haoran Xu et al. show in Offline RL with No OOD Actions: In-Sample Learning via Implicit Value Regularization that in-sample learning can be mathematically framed as solving a regularized Markov Decision Process. By utilizing implicit value regularization, the optimal policy can be derived without ever generating out-of-distribution actions.
This support-constrained paradigm is highly practical for safety-critical systems. For instance, Shengpu Tang et al. utilize batch-constrained policy optimization in Exploring Time-Step Size in Reinforcement Learning for Sepsis Treatment to ensure that clinical recommendations strictly adhere to high-probability actions in historical patient data, minimizing dangerous extrapolation errors.
Balancing Conservatism with Value-Guided Exploration
A natural tension arises in offline RL: if the agent is entirely constrained to the dataset, how can it ever outperform the behavior policy that collected it?
The answer lies in selective, value-guided extrapolation. By combining a conservative value baseline with a forward-looking model, agents can safely identify high-reward trajectories hidden within sub-optimal data. For example, Jingtong Gao et al. demonstrate in Generative Auto-Bidding with Value-Guided Explorations that using a learned value function to guide sequence modeling allows an agent to safely discover superior bidding strategies from static, sub-optimal datasets without triggering OOD instability. By grounding the exploration process in a well-regularized value function, the agent extracts the best possible policy from the data while remaining insulated from catastrophic distribution shifts.
Key Foundational Work
Engagement Rewarded Actor-Critic with Conservative Q-Learning for Speech-Driven Laughter Backchannel Generation (2021) — contributed 1 concept elements to this direction.
A General Offline Reinforcement Learning Framework for Interactive Recommendation (2021) — contributed 1 concept elements to this direction.
Understanding the Effects of Dataset Characteristics on Offline Reinforcement Learning (2021) — contributed 1 concept elements to this direction.