RMLGym: a Formal Reward Machine Framework for Reinforcement Learning

Hisham Unniyankal, Francesco Belardinelli, Angelo Ferrando, Vadim MalvoneWOA 2023

Introduction

Designing effective reward functions is one of the most notorious "dark arts" in reinforcement learning (RL). If a reward is too simple, the agent might find a loophole (reward hacking); if it’s too complex, the agent may never learn at all. This is known as the alignment problem—the discrepancy between what we want the agent to do and what we actually incentivize it to do.

RMLGym is a novel framework that bridges the gap between RL and Runtime Verification (RV) by using formal logic to automate reward design. By integrating the Runtime Monitoring Language (RML) with OpenAI Gym, researchers can now define complex, safety-critical agent behaviors through high-level specifications rather than tedious manual reward engineering.

The RMLGym Framework

The authors introduce RMLGym, a tool that wraps the standard OpenAI Gym interface and connects it to an RML Monitor. The architecture is designed to be modular and algorithm-agnostic.

The RMLGym architecture, illustrating the interaction between the Agent, the RMLGym Environment, and the RML Reward Monitor.

The system consists of three primary components:

  1. The Agent: A standard RL algorithm (such as PPO, SAC, or TD3). Because the framework is algorithm-agnostic, the agent does not need to know that rewards are generated by a formal monitor.
  2. RMLGym Environment: An extension of the OpenAI Gym environment. It utilizes a YAML configuration file to map environment states to formal variables used in the specification.
  3. RML Reward Constructor & Monitor: At each timestep, the environment sends state data to the RML Monitor via WebSockets. The monitor evaluates this data against a formal RML specification and returns a verdict based on a four-valued logic (True, Currently True, Currently False, or False). The constructor then translates these logical verdicts into numerical rewards (e.g., +1 for satisfaction, -1 for violation).

RML is particularly powerful because it can specify non-context-free properties and parametric specifications, offering greater expressivity than the temporal logics (like STL) used in prior frameworks.

Evaluation and Benchmarking

The framework was evaluated using the Pendulum-v1 environment from OpenAI Gym. This classic control benchmark requires swinging a pendulum into an upright position and balancing it using limited torque. It serves as a primary testbed for measuring the efficiency of RML-generated rewards.

Key Results

The study compared RML-based rewards against standard OpenAI Gym baselines and STL-based rewards:

  • Learning Efficiency: RML-based rewards enabled the PPO algorithm to reach an average return of 130 in approximately 250,000 timesteps, outperforming the baseline reward which required 300,000 timesteps.
  • Algorithm Performance: When utilizing RML rewards, off-policy algorithms like SAC and TD3 significantly outperformed PPO, converging to high rewards in fewer than 20,000 timesteps.
  • Robustness: RML rewards proved more effective than "sparse" STL rewards, which often suffer from the "vanishing gradient" problem where agents receive no feedback until the end of an episode.

Insights and Future Directions

RMLGym demonstrates that formal logic can serve as a "Reward Machine." Instead of relying on manual mathematical formulas, a formal monitor acts as the ground truth, ensuring the agent's performance is strictly aligned with the intended specification. This decoupling of logic from physics allows researchers to swap specifications without modifying the underlying environment code.

Currently, RMLGym relies on Boolean-style verdicts. Future work aims to incorporate quantitative rewards to measure the degree of satisfaction or violation, providing a smoother gradient for learning. Additionally, the authors plan to extend compatibility beyond OpenAI Gym to more complex, non-Markovian simulation environments.

Summary updated

BibTeX
@inproceedings{unniyankal2023rmlgym,
  author = {Hisham Unniyankal and Francesco Belardinelli and Angelo Ferrando and Vadim Malvone},
  title = {{RMLGym: a Formal Reward Machine Framework for Reinforcement Learning}},
  booktitle = {WOA},
  year = {2023},
}
Figure