Inferring Environment Dynamics from Expert Actions

This direction is split into other directions: pragmatic reward inference. Its last version stays readable and citable here.

Version 1 of 113 papers22 conceptsfrozen 2026-10-04

In traditional apprenticeship learning, we watch an expert to figure out what they want. We assume we already understand the "physics" of the environment—the transition dynamics—and use the expert's choices to back-calculate their reward function.

However, a growing body of research suggests that an expert’s actions are just as informative about how the world works as they are about what the expert values. If a master sailor suddenly adjusts their grip, it isn't just because they want to reach a destination; it’s because they perceive a change in the wind that a novice cannot yet see. By treating expert behavior as a window into the environment's underlying structure, agents can learn transition models and hidden states far more efficiently than by simple trial and error.

The Expert as a Mirror of Dynamics

The core intuition behind this research direction is that an optimal policy is inherently conditioned on the environment's dynamics. As Michael Herman, Tobias Gindele, and colleagues argue in their work on SERD, a policy acts as a "mirror" of the dynamics. If we assume an expert is acting rationally, every action they take is a bet on a specific outcome.

When an agent observes an expert choosing action $A$ over action $B$, it isn't just seeing a preference; it is seeing a structural constraint. Abhishek Sharma and Finale Doshi-Velez formalize this in Inverse Transition Learning (ITL). Instead of treating expert trajectories as isolated data points, ITL treats them as evidence that the chosen action leads to a higher-value state than the alternatives. This allows a learner to prune the space of possible transition functions, effectively "filling in" the physics of the world even in regions where the agent has never personally ventured.

Bayesian Inference of Latent Physics

If we view the expert as a highly calibrated sensor, we can use Bayesian updates to refine our world model. Botao Hao, Tor Lattimore, and Benjamin Van Roy demonstrate this with Informed Thompson Sampling (iTS). By incorporating the likelihood of an expert's action into the posterior distribution of the environment's parameters, the agent can infer which "arms" of a bandit are sub-optimal without ever pulling them.

This Bayesian perspective also helps resolve discrepancies between the learner and the expert. Ran Wei and Alfredo Garcia propose a framework that simultaneously estimates rewards and the expert's "subjective" model. This accounts for the fact that an expert might act based on a more sophisticated understanding of physics than the learner possesses. By modeling this internal "subjective" world, the learner becomes robust to distribution shifts, as it understands why the expert made a choice based on the underlying mechanics of the task.

State Inference and Cognitive Maps

In partially observable environments, the challenge isn't just learning the transitions, but identifying the current state itself. Takaki Makino and Johane Takeuchi showed that expert actions serve as informative signals for recovering both transition and observation models. Because the expert acts based on their internal "belief" of the state, their actions reveal that belief to the observer.

This mirrors how biological systems operate. Shannon Sandra Schiereck and Cristina Savin highlight that expertise facilitates a transition to active state inference. Instead of passively tracking rewards, experts use "cognitive maps" to recognize when environmental rules have changed. For a learner, observing an expert allows them to adopt these maps, enabling them to leapfrog the slow process of evidence accumulation and react to environmental shifts after a single informative event.

The Curse and Blessing of Expertise

While expertise is a powerful signal, it can be double-edged. Geng Zhao and Michael I. Jordan identify a "curse of expertise" in Stackelberg games, where an expert's perfect responses can mask the consequences of a leader's bad actions, slowing down learning.

The solution, however, remains the same: treat the expert's response as side information. By explicitly modeling the expert as an informed partner, the learner can infer constraints on the environment that are not visible through rewards alone. Whether in robotics or healthcare, the shift from "what are they doing?" to "what do they know that I don't?" allows agents to build more accurate, safer, and more adaptable models of the world.

Papers in this version

History & citationVersion 1 · 2026-10-04

Split into other directions · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_inferring_environment_dynamics_from_expe_d5093d38,
  title = {Inferring Environment Dynamics from Expert Actions},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/inferring-environment-dynamics-from-expert-actions/rdr_d5093d388b0e4c03a120db7bb755d86d},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_inferring_environment_dynamics_from_expe_d5093d38_v1,
  title = {Inferring Environment Dynamics from Expert Actions},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/inferring-environment-dynamics-from-expert-actions/rdr_d5093d388b0e4c03a120db7bb755d86d/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}