Programmatic Reward Design by Example

Weichao Zhou, Wenchao Li

Abstract

Reward design is a fundamental problem in reinforcement learning (RL). A misspecified or poorly designed reward can result in low sample efficiency and undesired behaviors. In this paper, we propose the idea of programmatic reward design, i.e. using programs to specify the reward functions in RL environments. Programs allow human engineers to express sub-goals and complex task scenarios in a structured and interpretable way. The challenge of programmatic reward design, however, is that while humans can provide the high-level structures, properly setting the low-level details, such as the right amount of reward for a specific sub-task, remains difficult. A major contribution of this paper is a probabilistic framework that can infer the best candidate programmatic reward function from expert demonstrations. Inspired by recent generative-adversarial approaches, our framework searches for the most likely programmatic reward function under which the optimally generated trajectories cannot be differentiated from the demonstrated trajectories. Experimental results show that programmatic reward functions learned using this framework can significantly outperform those learned using existing reward learning algorithms, and enable RL agents to achieve state-of-the-art performance on highly complex tasks.

Introduction

In Reinforcement Learning (RL), the reward function is the primary signal that guides agent behavior. However, designing effective reward functions is notoriously difficult. Sparse rewards (e.g., +1 only upon task completion) often lead to inefficient exploration, while dense, manually-shaped rewards are prone to "reward hacking," where agents exploit logic flaws to maximize score without achieving the intended goal. Inverse Reinforcement Learning (IRL) attempts to learn rewards from expert demonstrations, but typically produces black-box neural network models that lack interpretability and are difficult to constrain with human logic.

This paper introduces Programmatic Reward Design by Example (PRDBE), a framework that combines the structural domain knowledge of human engineers with data-driven optimization. By allowing humans to provide a high-level "sketch" of a reward function as a readable program, PRDBE enables the automatic inference of low-level numerical details from expert demonstrations.

Methodology

PRDBE leverages the concept of Program Synthesis by Sketching. A human designer writes a reward function using a Domain Specific Language (DSL), outlining the high-level logic (e.g., rewarding key pickup or penalizing door closing) while leaving specific numerical values as "holes" (?). This approach allows for the injection of memory and hindsight—checking if an agent has previously performed an action—which is difficult to achieve with standard Markovian rewards.

A programmatic reward sketch (b, d) and its corresponding state machine representation (c) for a grid-world environment (a).

To fill these holes, the system uses a generative adversarial framework. A discriminator attempts to distinguish expert trajectories from those generated by an RL agent. The "generator" is a neural network sampler that proposes numerical values for the program's holes. Because programs are non-differentiable, the authors optimize the hole-sampler to maximize an Evidence Lower Bound (ELBO) using self-normalized importance sampling. This avoids the need to fully train an RL agent for every candidate program, significantly improving efficiency.

Additionally, designers can specify symbolic constraints (e.g., ensuring the goal reward is greater than any sub-goal reward) to explicitly prevent reward hacking and guide the optimization process.

Experimental Evaluation

The framework was evaluated on the MiniGrid benchmark, focusing on three procedurally generated, complex exploration environments:

  • DoorKey: Requires finding a key and unlocking a door to reach a goal.
  • KeyCorridor: Involves searching multiple rooms for a key to unlock a specific door containing a target object.
  • ObstructedMaze: A complex maze where doors are blocked by objects and keys are hidden in boxes.

Key Results and Insights

  • State-of-the-Art Performance: Agents trained with PRDBE-completed rewards successfully solved complex tasks where standard IRL baselines (like GAIL and GAN-GCL) failed completely.
  • Extreme Sample Efficiency: PRDBE can infer effective reward functions from as few as 1 to 10 expert trajectories, a significant improvement over purely data-driven methods.
  • Zero-Shot Generalization: Because the learned rewards are logical programs, they generalize across environment scales. A reward learned on an $8 \times 8$ grid was successfully used to train agents on a $16 \times 16$ grid without further tuning.
  • Interpretability and Safety: The programmatic nature of the rewards allows humans to inspect the learned logic and enforce safety through symbolic constraints, ensuring the agent does not get stuck in sub-optimal loops.

Conclusion

PRDBE represents a significant advancement in reward design by allowing humans to define the "what" (logic) while the machine determines the "how much" (numerical values). While the current framework requires manual sketch creation, it provides a robust, interpretable, and sample-efficient path toward solving complex RL tasks. Future work will focus on automating the induction of the program structure itself to further reduce human effort.

Summary updated

Abstract

Reward design is a fundamental problem in reinforcement learning (RL). A misspecified or poorly designed reward can result in low sample efficiency and undesired behaviors. In this paper, we propose the idea of programmatic reward design, i.e. using programs to specify the reward functions in RL environments. Programs allow human engineers to express sub-goals and complex task scenarios in a structured and interpretable way. The challenge of programmatic reward design, however, is that while humans can provide the high-level structures, properly setting the low-level details, such as the right amount of reward for a specific sub-task, remains difficult. A major contribution of this paper is a probabilistic framework that can infer the best candidate programmatic reward function from expert demonstrations. Inspired by recent generative-adversarial approaches, our framework searches for the most likely programmatic reward function under which the optimally generated trajectories cannot be differentiated from the demonstrated trajectories. Experimental results show that programmatic reward functionslearned using this framework can significantly outperform those learned using existing reward learning algo-rithms, and enable RL agents to achieve state-of-the-artperformance on highly complex tasks.
BibTeX
@inproceedings{zhou2022programmatic,
  author = {Weichao Zhou and Wenchao Li},
  title = {{Programmatic Reward Design by Example}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2022},
  url = {https://openreview.net/forum?id=V9xBDBueEG},
}
Figure