Introduction
Reinforcement Learning (RL) agents typically learn through trial and error, which poses significant risks in safety-critical domains like autonomous driving or industrial control. While action masking is a common technique to prevent unsafe actions, most implementations are Markovian—they only consider the current state. Pure-Past Action Masking (PPAM) addresses this by providing a lightweight framework for enforcing non-Markovian safety constraints based on the system's history.
The Challenge of Non-Markovian Safety
Many real-world safety rules are temporal: "do not perform action A if event B occurred within the last $N$ steps." Traditional methods like "shielding" can handle these constraints by synthesizing a safety monitor. However, shielding is computationally expensive, often requiring double-exponential complexity relative to the safety specification. PPAM offers a more efficient alternative that maintains the same expressive power while reducing computational overhead to a linear scale.
Pure-Past Action Masking (PPAM)
The PPAM framework relies on Pure-Past Linear Temporal Logic (PPLTL). Unlike standard LTL, which looks into the future, PPLTL uses operators like "yesterday" ($\ominus$) and "since" ($S$) to evaluate finite traces of past behavior.
As shown in Figure 1, the architecture decouples safety monitoring from the learning agent. A dedicated feature extractor tracks the system history ($w$) to evaluate PPLTL formulas for each action. If a formula is satisfied, the corresponding action is included in the "Allowed Actions" mask. The agent then selects an action $a$ from this restricted set. This "Separation of Concerns" ensures that the agent's state representation ($s$) can remain focused on the task, while the safety mask handles the temporal constraints.
Experimental Evaluation
The authors tested PPAM across three distinct environments:
- Hot Water Tank: A control task requiring the agent to maintain water levels without excessive valve switching.
- Cocktail Party: A grid-world robot task with complex social safety rules (e.g., not over-serving guests).
- BoatRace: A safety gridworld designed to test "reward gaming," where agents must complete laps in a specific order.
Results and Discussion
In all experiments, PPAM-enabled agents achieved zero safety violations during both training and deployment. Beyond safety, the masking of irrelevant or dangerous actions significantly improved sample efficiency.
In the Cocktail Party environment (Figure 2), PPAM reached a high median reward much faster than the "Restraining Bolt" (RB) baseline. While RB uses negative rewards to discourage violations, it does not physically block them, leading to slower convergence and potential safety breaches during training.
In the BoatRace environment, PPAM successfully prevented "reward gaming." While a vanilla agent learned to oscillate between two checkpoints to accumulate points without finishing laps, the PPAM agent—constrained by history-based rules—learned to complete the full circuit correctly.
Conclusion
PPAM provides a robust and efficient framework for safe RL. By leveraging the backward-looking nature of PPLTL, it simplifies the enforcement of complex safety rules without the computational burden of traditional shielding. Future work will explore extending PPAM to continuous action spaces and partially observable environments.
Summary updated