Active Reward Inference through Environment Design

You are reading version 1 of this direction, frozen 2026-10-04. The latest version is version 2.

Version 1 of 212 papers22 conceptsfrozen 2026-10-04

In standard reinforcement learning (RL), an agent is usually given a reward function and told to maximize it. In inverse reinforcement learning (IRL), a learner observes an expert and tries to figure out what that reward function was. Both approaches share a common weakness: they are often passive. A passive learner must wait for the environment or the expert to provide informative data by chance. If an expert never visits a certain corner of the map, the learner can never know if that corner is dangerous or just boring.

A growing body of research suggests a more aggressive approach. Instead of waiting for data, the learner should actively "interrogate" the environment or the expert. By strategically changing the environment's rules or querying specific sequences of events, agents can resolve reward ambiguity and discover complex, history-dependent task logic significantly faster than through passive observation.

The Ambiguity of Passive Observation

The core problem in reward inference is ambiguity. As Thomas Kleine Buening and Christos Dimitrakakis point out in Environment Design for Inverse Reinforcement Learning, multiple reward functions can often explain the same observed behavior. If an expert walks around a puddle, is it because they hate water, or because they are taking the shortest path to a goal that happens to be on the other side?

In a static environment, you might never know. But if you are an Environment Designer, you can move the goal or place an obstacle in the expert's path. By observing how the expert responds to these changes, you can "nudge" them into revealing their true preferences. This concept, pioneered in early work by Haoqi Zhang and David C. Parkes in A General Approach to Environment Design with One Agent, treats the environment itself as a variable that can be optimized to elicit information.

Learning the Grammar of Rewards

Many real-world tasks are non-Markovian, meaning the reward depends on the history of actions (e.g., "pick up the key before opening the door"). To capture this, researchers use Reward Automata—finite-state machines that track the progress of a task.

Learning these machines passively is difficult because the agent might never stumble upon the specific sequence of events that triggers a state transition. To solve this, Maor Gaon and Ronen I. Brafman proposed integrating automata learning directly into the RL loop in Reinforcement Learning with Non-Markovian Rewards.

The mechanism often relies on the $L^*$ algorithm, a classic formal languages technique. In this framework, the learner acts as a student asking two types of questions:

  1. Membership Queries: "If I perform this specific sequence of actions, do I get a reward?"
  2. Equivalence Queries: "Is this state machine I've built a perfect map of the reward rules?"

In Active Finite Reward Automaton Inference and Reinforcement Learning Using Queries and Counterexamples, Zhe Xu and colleagues demonstrate that this "active interrogation" allows agents to solve complex tasks up to 88% faster than baselines.

The Agent as a Statistical Sensor

A key insight in this field is the dual role of the RL agent. In papers like Inferring Probabilistic Reward Machines from Non-Markovian Reward Signals for Reinforcement Learning, Taylor Dohmen and Alvaro Velasquez describe the agent as a statistical sensor or teacher.

When the $L^*$ algorithm needs to know if a specific trace of events results in a reward, it doesn't just guess. It treats that query as a sub-task for the RL agent. The agent is dispatched to "test" that specific hypothesis in the environment. If the agent's experience contradicts the current reward model, that experience serves as a counterexample that forces the model to be refined. This creates a virtuous cycle: better reward models lead to better policies, and better policies allow the agent to explore the environment more effectively to find further counterexamples.

Optimism and Efficiency

The ultimate goal of these methods is sample efficiency—learning the "rules of the game" in as few episodes as possible. Thomas Rupf and Andreas Krause take this to the extreme in Optimistic Task Inference for Behavior Foundation Models. By applying "optimism under uncertainty"—a principle from bandit theory—their agent specifically visits states that are most likely to resolve ambiguity about the reward function. This allows foundation models to master entirely new tasks in as few as five interaction episodes.

By shifting from passive observation to active environment design and interrogation, these researchers have turned reward inference from a game of chance into a systematic process of elimination. Whether through formal automata queries or strategic manipulation of environment dynamics, the agent no longer just learns within the world—it learns by challenging it.

Papers in this version

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 10 papers
  2. Version 1 · 2026-10-04 · 12 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_query_based_reward_learning_109e34fd,
  title = {query-based reward learning},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/query-based-reward-learning/rdr_109e34fdb9124a52be7a57895565c593},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-22},
}

This exact version (version 1, frozen text):

@misc{lacuna_active_reward_inference_through_environm_109e34fd_v1,
  title = {Active Reward Inference through Environment Design},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/active-reward-inference-through-environment-design/rdr_109e34fdb9124a52be7a57895565c593/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-22},
}