Abstract
Providing Reinforcement Learning agents with expert advice can dramatically improve various aspects of learning. Prior work has developed teaching protocols that enable agents to learn efficiently in complex environments; many of these methods tailor the teacher's guidance to agents with a particular representation or underlying learning scheme, offering effective but specialized teaching procedures. In this work, we explore protocol programs, an agent-agnostic schema for Human-in-the-Loop Reinforcement Learning. Our goal is to incorporate the beneficial properties of a human teacher into Reinforcement Learning without making strong assumptions about the inner workings of the agent. We show how to represent existing approaches such as action pruning, reward shaping, and training in simulation as special cases of our schema and conduct preliminary experiments on simple domains.
Introduction
Reinforcement learning (RL) agents often require human guidance to learn safely and efficiently, particularly in high-stakes real-world environments like autonomous driving or robotics. However, most existing methods for human-AI teaching are agent-specific. For instance, a teaching protocol might be hardcoded to modify the Q-values of a specific tabular Q-learning agent. This creates a significant bottleneck: upgrading an agent to a newer, more complex algorithm (e.g., from tabular Q-learning to Deep RL) necessitates a complete redesign of the human-guidance protocol.
This paper addresses this challenge by introducing "protocol programs," a framework that treats the learning agent as a complete black box. By intercepting and modifying the states, actions, and rewards passed between the agent and the environment, developers can implement safety constraints and learning accelerants across any RL algorithm without altering the agent's underlying code.
The Protocol Program Framework
The core of the proposed approach is the Protocol Program $P$, an agent-agnostic interface that acts as a middleware layer between the Environment $M$, the Agent $L$, and the Human teacher $H$. Because the protocol controls the communication channels, it can manipulate the agent's learning process purely through its inputs and outputs.
This simple framework captures several powerful teaching paradigms:
- Action Pruning (Safe Exploration): If the agent attempts a catastrophic action, the protocol intercepts it. Instead of sending the action to the real environment, the protocol blocks it, feeds the agent a large negative reward, and returns the agent to its previous state (a self-loop). This creates "invisible walls" that the agent learns to avoid naturally.
- Reward Shaping: The protocol can dynamically alter the reward signal based on human feedback to guide the agent toward desirable behaviors.
- Training in Simulation: The protocol can route the agent's actions to a simulated environment, only connecting it to the real world once the human deems the agent "ready."
The authors provide a theoretical guarantee for this approach: if a human teacher has even a roughly accurate understanding of the environment (a $\beta$-optimal Q-function), a protocol can be designed that never accidentally prunes the optimal action, while strictly guaranteeing the agent never takes an action worse than $4\beta$ below optimal.
Experimental Evaluation
The framework was evaluated on two modified RL benchmark environments: Catcher (a simplified Pong variant) and the Taxi Domain (a classic grid-world problem).
In the Catcher environment, the authors implemented a "speed limit" where moving the paddle too fast resulted in a massive negative reward. Using action pruning with a Trust Region Policy Optimization (TRPO) agent, the protocol-assisted agent completely avoided catastrophic high-speed actions. While the unassisted baseline eventually learned to avoid these actions, its total returns over 400,000 training steps were approximately five times worse than the assisted agent.
In the Taxi domain, the protocol pruned the dropoff action whenever the taxi was not at the correct destination. This drastically reduced the effective state space. Consequently, an R-MAX agent solved the problem in a tiny fraction of the episodes compared to a baseline, and standard Q-learning saw significant speedups.
Discussion and Future Work
The primary advantage of protocol programs is their agent-agnostic nature, allowing for the implementation of complex guidance strategies without knowledge of the agent's internal architecture. However, this black-box approach means the human cannot provide advice tailored to the agent's specific priors or exploration strategy. Furthermore, scaling action pruning to high-dimensional spaces remains a challenge, as it requires training supervised classifiers to recognize "catastrophes" from human labels, which can be difficult due to delayed consequences and non-stationary state-action distributions.
Future research directions include exploring dynamic state manipulation—where a human incrementally highlights or obscures features—and extending the framework to multi-agent settings. The authors also propose investigating "centaur" systems, where humans and AI dynamically share decision-making responsibilities in real-time.
Summary updated