Abstract
We study safe policy improvement (SPI) for partially observable Markov decision processes (POMDPs). SPI is an offline reinforcement learning (RL) problem that assumes access to (1) historical data about an environment, and (2) the so-called behavior policy that previously generated this data by interacting with the environment. SPI methods neither require access to a model nor the environment itself, and aim to reliably improve the behavior policy in an offline manner. Existing methods make the strong assumption that the environment is fully observable. In our novel approach to the SPI problem for POMDPs, we assume that a finite-state controller (FSC) represents the behavior policy and that finite memory is sufficient to derive optimal policies. This assumption allows us to map the POMDP to a finite-state fully observable MDP, the history MDP. We estimate this MDP by combining the historical data and the memory of the FSC, and compute an improved policy using an off-the-shelf SPI algorithm. The underlying SPI method constrains the policy-space according to the available data, such that the newly computed policy only differs from the behavior policy when sufficient data was available. We show that this new policy, converted into a new FSC for the (unknown) POMDP, outperforms the behavior policy with high probability. Experimental results on several well-established benchmarks show the applicability of the approach, even in cases where finite memory is not sufficient.
Introduction
In many real-world scenarios—such as medical treatment planning or industrial maintenance—improving an existing strategy (the "behavior policy") using historical logs is critical. This is the core challenge of Offline Reinforcement Learning. However, two major hurdles persist: safety and partial observability.
Safety requires a guarantee that a new policy will not perform significantly worse than the current one. Partial observability acknowledges that real-world sensors are often noisy and information is hidden, requiring agents to utilize memory for effective decision-making. While existing Safe Policy Improvement (SPI) methods typically assume perfect environmental visibility (MDPs), this paper introduces a framework for SPI in partially observable environments (POMDPs) by representing policies as Finite-State Controllers (FSCs).
Methodology: Bridging POMDPs and Safe RL
The proposed approach bridges the gap between complex POMDPs and safe MDP algorithms through a three-step process:
- Finite-State Controllers (FSCs): The behavior policy is represented as an FSC, which uses a set of "memory nodes" to track relevant past information.
- The History MDP: By assuming a finite window of history is sufficient for optimal decisions (stochastic bisimulation), the authors map the POMDP into a "finite-history MDP." In this space, states are combinations of memory nodes and current observations.
- SPIBB Integration: Once cast as an MDP, the authors apply Safe Policy Improvement with Baseline Bootstrapping (SPIBB). This pessimistic algorithm only alters the policy in areas where historical data is abundant. In data-scarce regions, it "bootstraps" back to the behavior policy to avoid unproven, risky actions.
Experimental Setup
The framework was evaluated on three classic POMDP benchmarks:
- CheeseMaze: A navigation task where the agent sees only nearby walls.
- Tiger: A problem requiring the agent to listen to noisy growls to locate a tiger behind two doors.
- Voicemail: A dialogue management task where a system must save or delete messages based on noisy user input.
These environments test the algorithm's robustness, as Tiger and Voicemail strictly violate the theoretical assumption that finite history is sufficient for optimal decision-making.
Key Results and Insights
The experiments yielded several critical findings regarding the reliability of SPI in partially observable settings:
- Reliability over Basic RL: Standard offline RL ("Basic RL") frequently suffered from catastrophic failures on small datasets, performing significantly worse than the baseline. In contrast, SPIBB consistently maintained or improved upon the behavior policy's performance.
- Safety Metrics: Using CVaR (Conditional Value at Risk) to measure worst-case outcomes, SPIBB maintained much higher safety margins than Basic RL, proving it is significantly less likely to deploy a failing policy.
- Memory Utility: Increasing the memory size ($k$) of the improved policy generally led to more reliable performance. In the Voicemail task, policies with more memory could reliably improve upon memoryless behavior policies even with limited data.
Conclusion and Future Work
The research demonstrates that treating an agent's internal memory as part of the environment's state allows standard RL tools to solve complex POMDP problems safely. While the method currently relies on exact matches in history (bisimulation), future work may explore distance metrics to group similar histories, enhancing scalability for high-dimensional or continuous observation spaces. Additionally, relaxing the requirement for the behavior policy to be an FSC could enable the use of more complex neural-network-based policies.
Summary updated