In standard reinforcement learning (RL), an agent treats the environment's reward signal as a black box. The agent takes actions, receives rewards, and updates its policy. However, this passive approach breaks down under two common real-world conditions: when rewards depend on a complex history of events (non-Markovian rewards), and when we must infer a hidden reward function from an agent's behavior (Inverse Reinforcement Learning, or IRL).
In both settings, relying on passive observation or stochastic exploration is highly inefficient. If a reward requires a precise sequence of historical steps, an agent might never stumble upon it by chance. Similarly, if we try to learn an expert's preferences just by watching them in a static environment, we face severe mathematical ambiguity—multiple distinct reward functions can explain the exact same behavior.
To solve these challenges, a growing body of research shifts the paradigm from passive observation to active interrogation. Instead of waiting for the environment or an expert to reveal reward structures, the learning system actively queries the environment or strategically alters its dynamics to force the underlying reward logic to reveal itself.
Active Interrogation via Automata Learning
Consider a task where an agent only receives a reward if it preheats an oven, cracks eggs, mixes flour, and then bakes a cake. Because the reward depends on a history of events, it is non-Markovian. We can represent this history-dependent logic using a state machine called a Finite Reward Automaton. But how can an agent efficiently discover the structure of this machine?
Instead of relying on random exploration to hit the exact sequence, the agent can treat the environment as a teacher and actively ask questions. This is typically operationalized using Angluin’s $L^*$ active learning algorithm, which builds an observation table of reward rules through two types of queries:
- Membership Queries: The learning framework asks, "What reward do I get if I execute exactly sequence $A \rightarrow B \rightarrow C$?" The agent then acts as a physical probe, executing that specific sequence in the environment to record the outcome.
- Equivalence Queries: The framework posits a hypothesis: "I think the reward automaton looks like Machine $M$. Is this correct?" The agent then explores the environment to find counterexamples where the real world deviates from Machine $M$'s predictions.
+-------------------------------------------------+
| |
| Learning Framework |
| (Maintains Hypothesis Automaton) |
| |
+--------+-------------------------------+---------+
| ^
| Membership / | Counterexamples /
| Equivalence Queries | Empirical Data
v |
+--------+-------------------------------+---------+
| |
| RL Agent / Sensor |
| (Executes Targeted Trajectories) |
| |
+------------------------+------------------------+
|
| Actions & Feedback
v
+------------+------------+
| |
| Environment |
| |
+-------------------------+
This active loop drastically accelerates learning. Xu et al. demonstrated that this active interrogation allows agents to master non-Markovian tasks up to 88% faster than passive baselines. Similarly, Rens et al. showed that treating the environment as a teacher allows an agent to systematically construct a minimal model of hidden reward logic without relying on stochastic luck.
Even when the environment is stochastic, this active framework holds. Dohmen et al. extended this concept to Probabilistic Reward Machines, where the RL agent acts as a statistical sensor. The agent explores the state space specifically to validate or refute the current reward model hypothesis, simultaneously optimizing its policy while gathering the precise empirical data needed to refine the reward logic. Gaon et al. further integrated this into the core RL loop, enabling agents to automatically augment their own state space with memory-providing automata through targeted history queries.
Environment Design as a Disambiguation Tool
Active reward inference is equally powerful when the goal is to learn a human's or an expert agent's hidden preferences. In standard Inverse Reinforcement Learning (IRL), a learner watches demonstrations and tries to deduce the underlying reward function. However, if the expert is observed in only one static environment, the learner cannot distinguish whether a behavior is driven by the expert's true internal goals or by external environmental constraints.
To resolve this ambiguity, the learner can step into the role of an environment designer. By systematically altering the environment's transition dynamics, rules, or layouts, the learner forces the expert to adapt, revealing their true underlying preferences.
Imagine trying to determine if a driver prefers the fastest route or the most scenic route. If both routes overlap on a single highway, observing them tells you nothing. But if you block the highway (altering the environment dynamics), the driver's detour immediately exposes their true priority.
The mathematical foundation for this approach was laid by Zhang et al., who showed that a designer can narrow down an agent's internal preferences by observing how they respond to environmental modifications.
Passive IRL: [ Expert Trajectory ] -------------------------> Multiple Ambiguous Rewards
(Overfits to static layout)
Active IRL: [ Modify Transitions/Rules ] -> [ Observe Response ] -> Unique Reward Identified
Applying this to cooperative settings, Kleine Buening et al. (2022) showed that an AI agent can uniquely identify a human partner's hidden goals by manipulating game transitions. Instead of passively watching the human play, the AI designs scenarios where different reward hypotheses would dictate completely distinct actions.
By proactively selecting a sequence of environments to challenge an expert, Kleine Buening et al. (2024) demonstrated that adaptive environment design successfully decouples an expert's true goals from environmental constraints (such as obstacles or moving targets). This active variation eliminates incorrect reward hypotheses far more robustly, requiring significantly fewer demonstrations than passive IRL.
Grounding Feedback and Optimistic Exploration
The philosophy of active reward inference also extends to settings where explicit reward signals are entirely absent, or where agents must rapidly generalize to unseen tasks.
When an agent must learn from unaligned, multi-dimensional feedback (such as human brainwaves or gestures), purely unsupervised methods like clustering fail because they cannot inherently tell a "good" feedback cluster from a "bad" one. Xie et al. introduced Interaction-Grounded Learning to overcome this. By actively interacting with the environment, the agent evaluates the correlation between its actions and subsequent changes in feedback. This interaction loop allows the agent to ground its own reward signal, discovering what positive feedback looks like purely through the consequences of its choices.
Finally, when deploying large behavior foundation models to new environments, agents must infer the target task on the fly. Rather than relying on massive pre-labeled datasets, Rupf et al. framed task inference as a linear bandit problem solved via "optimism under uncertainty." The agent actively visits states that maximize information gain about the reward function. By deliberately targeting states that resolve reward ambiguity, the agent can infer and master entirely unseen tasks in as few as five interaction episodes.
Summary
Whether an agent is learning a complex history-dependent task or trying to decode human preferences, passive observation forces the learner to rely on luck or cope with severe ambiguity. Active reward inference solves this by turning the learner into an active participant. By leveraging automata queries to map out non-Markovian rules, or using environment design to force experts to reveal their preferences, these frameworks prove that the fastest way to understand a reward function is to actively challenge it.
Key Foundational Work
Active Finite Reward Automaton Inference and Reinforcement Learning Using Queries and Counterexamples (2020) — contributed 3 concept elements to this direction.
Online Learning of Non-Markovian Reward Models (2020) — contributed 2 concept elements to this direction.
Reinforcement Learning with Non-Markovian Rewards (2020) — contributed 1 concept elements to this direction.