Programmatic Reward Design by Example

Weichao Zhou, Wenchao Li

Abstract

Reward design is a fundamental problem in reinforcement learning (RL). A misspecified or poorly designed reward can result in low sample efficiency and undesired behaviors. In this paper, we propose the idea of programmatic reward design, i.e. using programs to specify the reward functions in RL environments. Programs allow human engineers to express sub-goals and complex task scenarios in a structured and interpretable way. The challenge of programmatic reward design, however, is that while humans can provide the high-level structures, properly setting the low-level details, such as the right amount of reward for a specific sub-task, remains difficult. A major contribution of this paper is a probabilistic framework that can infer the best candidate programmatic reward function from expert demonstrations. Inspired by recent generative-adversarial approaches, our framework searches for the most likely programmatic reward function under which the optimally generated trajectories cannot be differentiated from the demonstrated trajectories. Experimental results show that programmatic reward functions learned using this framework can significantly outperform those learned using existing reward learning algorithms, and enable RL agents to achieve state-of-the-art performance on highly complex tasks.

Introduction

Reinforcement Learning (RL) is notoriously sensitive to reward design. Most environments use "goal-driven" rewards (e.g., +1 for finishing, 0 otherwise), which are often too sparse for agents to learn complex sequences of actions efficiently. To solve this, researchers often turn to Inverse Reinforcement Learning (IRL) to "reverse engineer" rewards from expert demonstrations. However, traditional IRL often produces "black-box" reward functions, such as deep neural networks, that are difficult for humans to interpret or verify. Conversely, logic-based rewards (using LTL or Finite State Automata) are interpretable but tedious to design manually.

This paper introduces Programmatic Reward Design by Example (PRDBE), a framework that bridges this gap. It allows engineers to define reward functions using high-level "program sketches" while automatically inferring specific numerical values (the "holes") from expert demonstrations. By combining the interpretability of domain-specific languages (DSLs) with generative adversarial learning, PRDBE enables RL agents to solve complex, multi-stage tasks with high sample efficiency.

Methodology

The PRDBE framework consists of three main components designed to leverage human domain knowledge while automating parameter tuning.

Program Sketches and Symbolic Constraints

A human provides a partial program in a DSL that outlines sub-goals but leaves "holes" (?id) for numerical reward values. This provides a massive inductive bias: the agent does not need to discover what matters (e.g., picking up a key), only how much it matters relative to other events. Additionally, designers can specify symbolic constraints (e.g., ?5 + ?4 <= 0) to prevent "reward hacking," such as an agent repeatedly picking up and dropping a key to accumulate infinite reward.

Figure 2: Example of a MiniGrid environment (a) and its corresponding programmatic reward sketch (b) with numerical holes.

Generative Adversarial Learning

To fill the "holes," the framework uses a probabilistic approach inspired by Generative Adversarial Imitation Learning (GAIL). Because programs are non-differentiable, the authors employ self-normalized importance sampling. A neural network "sampler" suggests candidate programs, which are then evaluated based on how well an agent trained on them matches expert behavior. A discriminator attempts to distinguish between expert trajectories and agent trajectories, providing the signal to refine the reward values in the program sketch.

Figure 6: The PRDBE optimization loop, illustrating the interaction between the program sampler, agent policy optimization, and the discriminator.

Evaluation and Results

The framework was evaluated using the MiniGrid environment, focusing on tasks with complex logic like Door-Key, KeyCorridor, and the highly difficult ObstructedMaze. PRDBE was compared against state-of-the-art baselines including PPO, AGAC, GAIL, and GAN-GCL.

  • Performance and Efficiency: In the ObstructedMaze task, PRDBE was the only method to achieve a high success rate. It reached peak performance with as few as one to six expert demonstrations, whereas traditional imitation learning often requires hundreds.
  • Generalization: Because programmatic rewards are tied to logical events rather than coordinates, rewards learned in small environments (e.g., 8x8) allowed agents to solve much larger environments (e.g., 16x16) zero-shot.
  • Interpretability: Unlike neural reward functions, the learned programmatic rewards are human-readable, allowing engineers to verify that the inferred values satisfy all logical constraints and safety requirements.

Conclusion

PRDBE demonstrates that programmatic reward functions can significantly outperform traditional reward learning algorithms. By using human-provided sketches as an inductive bias, the framework achieves state-of-the-art performance on complex tasks while maintaining interpretability. Future work may explore automated sketch generation to further reduce the manual effort required in DSL design.

Summary updated

Abstract

Reward design is a fundamental problem in reinforcement learning (RL). A misspecified or poorly designed reward can result in low sample efficiency and undesired behaviors. In this paper, we propose the idea of programmatic reward design, i.e. using programs to specify the reward functions in RL environments. Programs allow human engineers to express sub-goals and complex task scenarios in a structured and interpretable way. The challenge of programmatic reward design, however, is that while humans can provide the high-level structures, properly setting the low-level details, such as the right amount of reward for a specific sub-task, remains difficult. A major contribution of this paper is a probabilistic framework that can infer the best candidate programmatic reward function from expert demonstrations. Inspired by recent generative-adversarial approaches, our framework searches for themost likely programmatic reward function under whichthe optimally generated trajectories cannot be differen-tiated from the demonstrated trajectories. Experimental results show that programmatic reward functions learned using this framework can significantly outperform those learned using existing reward learning algorithms, and enable RL agents to achieve state-of-the-art performance on highly complex tasks.
BibTeX
@inproceedings{zhou2022programmatic,
  author = {Weichao Zhou and Wenchao Li},
  title = {{Programmatic Reward Design by Example}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2022},
  url = {https://openreview.net/forum?id=V9xBDBueEG},
}
Figure