Safe Policy Improvement with Baseline Bootstrapping

34 concepts21 papers1 proposal

In traditional reinforcement learning (RL), agents learn through trial and error. They explore an environment, make mistakes, and update their strategies based on the rewards or penalties they receive. However, in high-stakes domains like healthcare, industrial control, or finance, the "error" in trial-and-error can be catastrophic.

To deploy RL safely in these real-world settings, we often turn to offline reinforcement learning, where an agent learns exclusively from a fixed, pre-recorded dataset generated by an existing historical policy (the baseline policy). But offline RL introduces a severe challenge: epistemic uncertainty. If the dataset contains little to no information about a specific state-action pair, a standard RL algorithm might overestimate its value due to distributional shift. The agent becomes over-optimistic, selects an unobserved action, and fails catastrophically when deployed.

To prevent this, a family of algorithms known as Safe Policy Improvement with Baseline Bootstrapping (SPIBB) introduces a simple yet powerful intuition: only deviate from the baseline policy when you are absolutely certain the new action is better; otherwise, default to the baseline.


The Safety Floor: Why Bootstrapping Works

The core mechanism of baseline bootstrapping is to establish a performance "safety floor." Instead of applying a uniform penalty to all uncertain states—which can make the agent overly conservative and stall learning—the agent explicitly splits the state-action space based on data density.

As demonstrated by Kimia Nadjahi et al. in Safe Policy Improvement with Soft Baseline Bootstrapping, the learning policy is constrained to precisely replicate the baseline policy's behavior in regions with insufficient data. In regions where data is abundant, the algorithm allows the policy to freely optimize and deviate from the baseline. This ensures that the newly learned policy's overall performance is mathematically guaranteed not to drop below that of the historical baseline.

If Data Density is High:
   └── Optimize action selection freely to maximize reward
If Data Density is Low:
   └── Fall back and copy the baseline policy's action

This fallback strategy provides a much more data-efficient path to safety than traditional importance sampling methods, as noted by Harsh Satija et al. in Multi-Objective SPIBB: Seldonian Offline Policy Improvement with Safety Constraints in Finite MDPs. By leveraging the historical behavior as a safe fallback, the agent avoids the need to accurately model the transition dynamics of completely unobserved regions.


Quantifying Uncertainty and Deferral

To implement this mechanism, the agent must reliably identify when to defer to the baseline.

One straightforward approach is count-based thresholding. For instance, Filippo Vannella et al. in Remote Electrical Tilt Optimization via Safe Reinforcement Learning monitor the exact frequency of state-action pairs in the historical dataset. If a pair appears fewer times than a predefined threshold, the agent immediately defaults to the known-safe baseline. Leo Benac formalizes this concept as "density-based policy deferral" in Decision-Point Guided Safe Policy Improvement, showing that restricting updates to these dense data regions allows for tighter theoretical guarantees without even requiring prior knowledge of the behavior policy.

However, calculating these safety bounds requires rigorous mathematical formulation. Early iterations of these frameworks suffered from a subtle theoretical flaw: they relied on potentially incorrect assumptions about future state visitation frequencies. In Safe Policy Improvement Approaches and their Limitations, Philipp Scholl et al. revealed that some soft bootstrapping variants were not provably safe under limited data. To fix this, Felix Dietrich et al. derived rigorous lower bounds on policy performance in Safe Policy Improvement Approaches on Discrete Markov Decision Processes, ensuring that any deviation from the baseline is strictly advantageous relative to the baseline's action-value function.

To make these theoretical bounds practical for real-world applications where data collection is expensive, Marnix Suilen et al. introduced a "two-successor" model transformation in More for Less: Safe Policy Improvement with Stronger Performance Guarantees. By collapsing complex environment transitions into binary bounds, they drastically reduced the amount of data required to start safely outperforming the baseline.


From Hard Fallbacks to Relative Pessimism

While early SPIBB methods relied on a binary switch between the baseline and the optimized policy, modern approaches frame this problem through the lens of relative pessimism. Instead of penalizing all uncertain actions globally, relative pessimism ensures that the learned policy specifically avoids performing worse than the reference baseline.

This is often implemented as an adversarial game. In Adversarially Trained Actor Critic for Offline Reinforcement Learning, Ching-An Cheng et al. introduced ATAC, which frames offline RL as a two-player Stackelberg game. By training an actor against an adversarial critic, the algorithm ensures relative improvement over the behavior policy.

This game-theoretic framing offers two massive advantages:

  1. Robustness to Hyperparameters: Standard offline RL is notoriously sensitive to hyperparameter tuning. By anchoring the optimization to a reference policy, frameworks like ARMOR, developed by Mohak Bhardwaj et al. in Adversarial Model for Offline Reinforcement Learning, maintain a strict safety margin even under poor hyperparameter choices or inaccurate value estimations.
  2. Optimal Convergence: As shown by Arnob Ghosh et al. in Adversarially Trained Weighted Actor-Critic for Safe Offline Reinforcement Learning, combining relative pessimism with weighted actor-critic architectures allows the agent to achieve optimal statistical convergence rates while requiring significantly less data coverage than previous state-of-the-art methods.

Overcoming the Black-Box and Scaling Up

For baseline bootstrapping to work in practice, researchers have had to solve several structural hurdles:

By anchoring exploration to a known or estimated baseline, these methods bridge the gap between theoretical RL and high-stakes deployment. Whether applied to environmental monitoring (Federico Bianchi et al. in Safe and Efficient Reinforcement Learning for Environmental Monitoring) or used to bootstrap online search algorithms like Monte Carlo Tree Search (Federico Bianchi et al. in Developing safe and explainable autonomous agents: from simulation to the real world), baseline bootstrapping ensures that an agent's quest for optimization never comes at the cost of safety.

Key Foundational Work

Safe Policy Improvement with an Estimated Baseline Policy (2019) — contributed 3 concept elements to this direction.
Safe Policy Improvement by Minimizing Robust Baseline Regret (2016) — contributed 3 concept elements to this direction.
Safe Policy Improvement with Soft Baseline Bootstrapping (2019) — contributed 2 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_baseline_bootstrapping_806f63ce,
  title = {baseline bootstrapping},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/baseline-bootstrapping/rdr_806f63ceaee34469992d61fdbc3bbeab},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_baseline_bootstrapping_806f63ce_v1,
  title = {baseline bootstrapping},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/baseline-bootstrapping/rdr_806f63ceaee34469992d61fdbc3bbeab/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake