Abstract
Safe Policy Improvement (SPI) aims to derive a new policy from offline data that is guaranteed to outperform a baseline policy with high probability. This paper identifies a theoretical flaw in the safety proofs of the popular Soft-SPIBB algorithms, revealing that they are not provably safe as previously claimed. To address this, the authors introduce Adv-Soft-SPIBB, a truly provably safe alternative, and Lower-Approx-Soft-SPIBB, a high-performing heuristic. Through evaluations on Random MDPs and the Wet Chicken benchmark, the study demonstrates that while policy-restriction methods offer superior risk-sensitive performance, existing theoretical safety bounds remain overly conservative, requiring impractical amounts of data for real-world application.
Introduction
In safety-critical industrial applications, such as gas turbine control or medical treatment planning, Reinforcement Learning (RL) agents cannot learn through trial and error. Instead, researchers utilize Offline (or Batch) RL, where agents learn exclusively from pre-recorded datasets. The primary challenge in this setting is Safe Policy Improvement (SPI): finding a new policy that is guaranteed, with high probability, to be at least as good as the current baseline policy. While "Soft-SPIBB" algorithms were previously proposed as a flexible solution with safety guarantees, this research investigates the validity of those mathematical claims and their practical utility.
Taxonomy of SPI Approaches
The authors categorize existing SPI approaches into two distinct families based on their mechanism for handling uncertainty in the data:
- Uncertainty as a penalty: Algorithms such as Reward-adjusted MDP (RaMDP) or DUIPI lower the estimated value of state-action pairs that are underrepresented in the training data.
- Uncertainty as a restriction: Algorithms like SPIBB limit the degree to which the new policy can deviate from the baseline policy in uncertain regions of the state space.
Addressing the Safety Gap
A critical contribution of this work is the mathematical disproof of the safety assumption used in prior Soft-SPIBB research. The original proofs relied on the assumption that the uncertainty of a future state is bounded by the uncertainty of the current state—a premise that does not hold for all Markov Decision Processes (MDPs). To rectify this, the authors propose two new methods:
- Adv-Approx-Soft-SPIBB: A refined version incorporating an "advantageous" constraint. It ensures the new policy only selects actions that appear superior to the baseline according to Monte-Carlo estimates, restoring theoretical safety.
- Lower-Approx-Soft-SPIBB: A heuristic that allows the agent to reduce the probability of taking "bad" uncertain actions without being penalized, mirroring the logic of the successful $\Pi_{\leq b}$-SPIBB algorithm.
Experimental Evaluation and Results
The algorithms were evaluated on two benchmarks: Random MDPs (50 states) and the Wet Chicken Benchmark, a complex environment featuring heterogeneous stochasticity that tests an algorithm's ability to handle varying levels of randomness.
Key Findings
- Policy Restriction vs. Value Penalty: While penalizing the value function (Group 1) often improves average performance, restricting the policy set (Group 2) is significantly more effective for risk-sensitive performance, specifically when measured by the 1%-Conditional Value at Risk (1%-CVaR).
- Top Performer: Lower-Approx-Soft-SPIBB emerged as the most robust algorithm, balancing high mean performance with strong safety across both benchmarks.
- The Data Gap: A significant disparity exists between theoretical safety and practical utility. For the Wet Chicken task, the standard SPIBB algorithm would require over 1.8 million observations to provide a meaningful safety bound for a simple 25-state problem, highlighting the extreme conservatism of current mathematical guarantees.
Conclusion and Future Work
While this work provides a more rigorous foundation for Safe Policy Improvement, it also highlights the limitations of current theoretical bounds. The "safety envelopes" provided by current mathematics are often too loose for practical engineering applications. Future research will focus on developing tighter error functions, such as those using Empirical Bernstein bounds, and extending these frameworks to continuous state spaces common in robotics and complex industrial systems.
Summary updated