Introduction
In offline reinforcement learning (RL), the goal is to derive an improved policy from a fixed dataset of historical interactions. A significant challenge in this domain is "policy collapse," where a newly learned policy performs worse than the original behavior policy due to insufficient data coverage. Safe Policy Improvement (SPI) algorithms address this by providing probabilistic guarantees that the new policy will at least match the performance of the baseline.
While SPI is well-studied for Markov Decision Processes (MDPs), real-world applications often involve Partially Observable MDPs (POMDPs), where agents lack full state visibility. This paper extends SPI to POMDPs by utilizing Finite-State Controllers (FSCs) to provide the necessary memory for effective decision-making in partially observable environments.
Methodology: From POMDPs to History MDPs
The authors propose a framework that transforms a POMDP into a tractable "History MDP." This transformation is achieved through two key components:
- Finite-State Controllers (FSCs): Unlike reactive policies, FSCs maintain an internal memory state (nodes) that updates based on incoming observations. This allows the agent to base decisions on a sequence of past events.
- History MDP Mapping: By assuming that a finite history window is sufficient for near-optimal behavior, the POMDP's history is mapped into the states of a new MDP. Each state in this History MDP is defined by the current FSC memory node and the most recent observation.
Safe Policy Improvement with Baseline Bootstrapping (SPIBB)
Once the History MDP is established, the authors apply the SPIBB algorithm. SPIBB is a pessimistic approach: it identifies state-action pairs with insufficient data in the buffer and "bootstraps" them by defaulting to the behavior policy. This prevents the agent from taking unverified risks in poorly explored regions of the state space. The authors provide a theoretical performance bound ($\zeta$) ensuring that the improved policy will outperform the behavior policy with high probability ($1-\delta$).
Experimental Evaluation
The framework was evaluated on three standard POMDP benchmarks: CheeseMaze, Tiger, and Voicemail. These environments test various aspects of partial observability, from spatial navigation with limited visibility to inferring intent from noisy signals.
Key Results
- Reliability: In all benchmarks, "Basic RL" (which solves the estimated model without safety constraints) frequently performed worse than the behavior policy when data was scarce. In contrast, SPIBB consistently met or exceeded the baseline performance.
- Data Efficiency: As the dataset size ($|\mathcal{D}|$) increased, SPIBB-derived policies converged toward the performance of PO-UCT, a powerful online solver. This indicates that the safety constraints do not hinder the discovery of near-optimal policies given sufficient data.
- Memory and Safety: A critical insight is that increasing the memory size ($k$) of the FSC not only improves potential performance but also enhances the reliability of the SPI guarantees.
Conclusion and Future Work
This research successfully bridges the gap between theoretical SPI guarantees and the practical complexities of POMDPs. By leveraging FSCs and History MDPs, the authors provide a robust method for offline policy improvement in partially observable settings. Future work may explore approximate bisimilarity to improve data efficiency and the integration of deep learning architectures to handle high-dimensional observation spaces.
Summary updated