Pure-Past Action Masking

Giovanni Varricchione, Natasha Alechina, Mehdi Dastani, Giuseppe De Giacomo, Brian Logan, Giuseppe PerelliAAAI 2024 · OpenReview

Introduction

Reinforcement Learning (RL) agents typically learn through trial and error, which poses significant risks in safety-critical domains like autonomous driving or industrial control. While action masking is a common technique to prevent unsafe actions, most implementations are Markovian—they only consider the current state. Pure-Past Action Masking (PPAM) addresses this by providing a lightweight framework for enforcing non-Markovian safety constraints based on the system's history.

The Challenge of Non-Markovian Safety

Many real-world safety rules are temporal: "do not perform action A if event B occurred within the last $N$ steps." Traditional methods like "shielding" can handle these constraints by synthesizing a safety monitor. However, shielding is computationally expensive, often requiring double-exponential complexity relative to the safety specification. PPAM offers a more efficient alternative that maintains the same expressive power while reducing computational overhead to a linear scale.

Pure-Past Action Masking (PPAM)

The PPAM framework relies on Pure-Past Linear Temporal Logic (PPLTL). Unlike standard LTL, which looks into the future, PPLTL uses operators like "yesterday" ($\ominus$) and "since" ($S$) to evaluate finite traces of past behavior.

The PPAM architecture separates safety feature extraction from agent learning, providing a set of allowed actions based on system history.

As shown in Figure 1, the architecture decouples safety monitoring from the learning agent. A dedicated feature extractor tracks the system history ($w$) to evaluate PPLTL formulas for each action. If a formula is satisfied, the corresponding action is included in the "Allowed Actions" mask. The agent then selects an action $a$ from this restricted set. This "Separation of Concerns" ensures that the agent's state representation ($s$) can remain focused on the task, while the safety mask handles the temporal constraints.

Experimental Evaluation

The authors tested PPAM across three distinct environments:

  1. Hot Water Tank: A control task requiring the agent to maintain water levels without excessive valve switching.
  2. Cocktail Party: A grid-world robot task with complex social safety rules (e.g., not over-serving guests).
  3. BoatRace: A safety gridworld designed to test "reward gaming," where agents must complete laps in a specific order.

Results and Discussion

In all experiments, PPAM-enabled agents achieved zero safety violations during both training and deployment. Beyond safety, the masking of irrelevant or dangerous actions significantly improved sample efficiency.

Comparison of median reward over time between PPAM and the Restraining Bolt (RB) baseline in the Cocktail Party environment.

In the Cocktail Party environment (Figure 2), PPAM reached a high median reward much faster than the "Restraining Bolt" (RB) baseline. While RB uses negative rewards to discourage violations, it does not physically block them, leading to slower convergence and potential safety breaches during training.

In the BoatRace environment, PPAM successfully prevented "reward gaming." While a vanilla agent learned to oscillate between two checkpoints to accumulate points without finishing laps, the PPAM agent—constrained by history-based rules—learned to complete the full circuit correctly.

Conclusion

PPAM provides a robust and efficient framework for safe RL. By leveraging the backward-looking nature of PPLTL, it simplifies the enforcement of complex safety rules without the computational burden of traditional shielding. Future work will explore extending PPAM to continuous action spaces and partially observable environments.

Summary updated

Abstract

We present Pure-Past Action Masking (PPAM), a lightweight approach to action masking for safe reinforcement learning. In PPAM, actions are disallowed (“masked”) according to specifications expressed in Pure-Past Linear Temporal Logic (PPLTL). PPAM can enforce non-Markovian constraints, i.e., constraints based on the history of the system, rather than just the current state of the (possibly hidden) MDP. The features used in the safety constraint need not be the same as those used by the learning agent, allowing a clear separation of concerns between the safety constraints and reward specifications of the (learning) agent. We prove formally that an agent trained with PPAM can learn any optimal policy that satisfies the safety constraints, and that they are as expressive as shields, another approach to enforce non-Markovian constraints in RL. Finally, we provide empirical results showing how PPAM can guarantee constraint satisfaction in practice.
BibTeX
@inproceedings{varricchione2024pure,
  author = {Giovanni Varricchione and Natasha Alechina and Mehdi Dastani and Giuseppe De Giacomo and Brian Logan and Giuseppe Perelli},
  title = {{Pure-Past Action Masking}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2024},
  url = {https://openreview.net/forum?id=3Dlvpmfhtv},
}
Figure