Learning Latent Rewards and Dynamics through Pragmatic Inference

18 concepts10 papers1 proposal

When an artificial agent learns from a human expert, it traditionally treats the expert's behavior as literal data. In imitation learning, an expert's action is a direct label to copy; in instruction following, a verbal command is a literal constraint to satisfy; in reinforcement learning (RL), human feedback is a simple scalar reward.

However, human communication and decision-making are fundamentally pragmatic. When a human trainer gives an instruction, chooses an action, or even remains silent, they do so intent-fully, choosing their behavior based on their internal knowledge of the environment and their underlying goals.

A growing body of research shifts the paradigm from literal interpretation to pragmatic inference. Instead of taking human input at face value, these frameworks treat human behavior—whether it is language, physical actions, or silence—as an observed signal generated by a latent cognitive process. By modeling the expert's intent and inverting this generative model via Bayesian inference, an agent can infer hidden reward functions, environmental transition dynamics, and operational horizons far more efficiently than through trial-and-error alone.


Inverting the Speaker: Language as a Window to Rewards

How can an agent truly understand what a human wants from a simple sentence? If a teacher says, "Take the red mushroom," a literal agent simply learns to execute that specific action in that specific state. A pragmatic agent, however, asks a deeper question: Why did the teacher choose that specific utterance over all other possibilities?

This intuition is formalized by applying the Rational Speech Acts (RSA) framework to sequential decision-making. As demonstrated by Theodore R. Sumers et al. in [How to talk so AI will learn: Instructions, descriptions, and autonomy] and [How to talk so AI will learn: instructions, descriptions, and pragmatics], we can model a human teacher as a speaker who selects an utterance $u$ to maximize the agent's expected utility under a latent reward function $R$. Because the teacher is modeled as rational, the agent can use Bayes' rule to invert this speaker model:

$$P(R \mid u) \propto P(u \mid R) P(R)$$

This inversion allows the agent to extract the underlying values associated with environmental features, rather than just copying actions. For instance, the authors show that by analyzing the structural type of language used—distinguishing between state-specific instructions ("take this path") and abstract descriptions ("swamps are dangerous")—the agent can jointly infer both the reward function and the intended horizon of autonomy. This enables the agent to calibrate exactly how broadly to generalize the human's preference to entirely new contexts.

Literal Agent:   Utterance ──> Execute Action
Pragmatic Agent: Utterance ──> Invert Speaker Model ──> Infer Latent Reward ──> Generalize to New States

By treating language as a dual signal containing both immediate guidance and long-term preference information, agents achieve remarkable robustness. As Jessy Lin et al. show in [Inferring Rewards from Language in Context], evaluating why a user chose a specific description over alternative available options resolves linguistic ambiguity. This pragmatic reasoning exposes the salient features of a user's reward function, allowing the agent to successfully navigate completely novel environments where literal instruction-following would fail.


Learning Something from Nothing: The Pragmatics of Silence

In traditional reinforcement learning, if an agent receives no feedback from a human trainer, it registers a reward of zero. This assumes silence is empty. In reality, human trainers use silence strategically, but their strategies vary: one trainer might only give positive reinforcement for good actions and remain silent otherwise, while another might only punish mistakes.

To exploit this, Bei Peng 0001 et al. developed a framework in [Learning something from nothing: Leveraging implicit human feedback strategies] that treats a trainer's feedback (or lack thereof) as a joint distribution of the target task and the trainer's latent feedback strategy. By running a simultaneous Bayesian inference over both variables, the agent dynamically decodes the meaning of silence. If the agent infers that the trainer utilizes a "reward-only" strategy, silence is correctly interpreted as implicit disapproval; if the trainer utilizes a "punishment-only" strategy, silence becomes a green light of implicit approval.

This concept of informative omission also applies to natural language feedback. In [Learning Rewards From Linguistic Feedback], Theodore Sumers et al. incorporate Gricean Maxims—the linguistic principle that speakers provide the most informative, relevant information without being redundant—into reward learning. If a teacher praises certain features of an agent's behavior but omits others, a pragmatic listener infers that the unmentioned features are likely less desirable or irrelevant. "Reading between the lines" allows the agent to reconstruct highly accurate reward functions from sparse, indirect linguistic inputs.


Actions as Side Information: Inferring Environment Dynamics

The power of pragmatic inference is not restricted to language; it applies equally to physical actions. When an expert interacts with an environment, their decisions are dictated by their internal beliefs about how the world works. Therefore, an expert's actions can reveal hidden properties of the environment itself.

Consider a partially observable Markov decision process (POMDP) where the transition and observation models are unknown. Traditional apprenticeship learning focuses solely on recovering the reward function, assuming the dynamics are given. However, Takaki Makino et al. demonstrate in [Apprenticeship Learning for Model Parameters of Partially Observable Environments] that if we assume an expert acts optimally according to the true environment model, their actions serve as a rich diagnostic stream for recovering environmental parameters. By observing the expert's choices, the agent can infer the latent transition and observation rules of the world much faster than it could by simply watching how the environment reacts to random actions.

Traditional:  Expert Action ───────────────────────────────> Infer Reward Function
Pragmatic:    Expert Action ──> Invert Expert Policy ──> Infer Latent Environment Dynamics & Rewards

This insight is highly effective for accelerating online exploration. In multi-armed bandit problems, Botao Hao et al. introduce Informed Thompson Sampling (iTS) in [Leveraging Demonstrations to Improve Online Learning: Quality Matters]. By incorporating a model of expert competence into the agent's Bayesian update, the learner treats the expert's choice of action as a deliberate signal about the environment's underlying structure. By analyzing which actions a competent expert avoids, the learner can infer that those arms are sub-optimal even without receiving explicit reward signals from them, drastically reducing cumulative regret.

Even when interacting with an omniscient partner in game-theoretic settings, treating partner actions as a secondary data stream provides critical side information. As shown by Geng Zhao et al. in [Online Learning in Stackelberg Games with an Omniscient Follower], an expert's perfect responses can sometimes mask the consequences of bad actions (a phenomenon termed the "curse of expertise"). However, explicitly designing algorithms to leverage the follower's actions as an indirect signal of latent environmental constraints fundamentally accelerates the leader's learning process. This perspective aligns with work by Finale Doshi-Velez et al. in [Nonparametric Bayesian Policy Priors for Reinforcement Learning], which utilizes nonparametric Bayesian priors to treat expert demonstrations as simultaneous evidence of both world dynamics and policy structures.


Theory of Mind: Closing the Loop

Pragmatic inference is ultimately a two-way street. If an artificial agent can accelerate its own learning by modeling the intent behind human behavior, it can also use a predictive model of a human listener to communicate more effectively.

This closed-loop application of Theory-of-Mind (ToM) is explored by Pei Zhou et al. in [I Cast Detect Thoughts: Learning to Converse and Guide with Intents and Theory-of-Mind in Dungeons and Dragons]. In this framework, a primary agent is tasked with guiding a human partner toward a specific goal. Instead of optimizing for mere linguistic fluency, the agent uses a simulated model of the listener's cognitive response as a reward function within an RL loop.

By optimizing its utterances to maximize the probability of a desired reaction from the listener, the agent anticipates how its words will influence the partner's subsequent actions. This mechanism prioritizes the pragmatic effect of communication, allowing the agent to strategically guide, influence, and cooperate with human partners in complex, collaborative environments.

Key Foundational Work

Apprenticeship Learning for Model Parameters of Partially Observable Environments (2012) — contributed 3 concept elements to this direction.
Nonparametric Bayesian Policy Priors for Reinforcement Learning (2010) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_pragmatic_reward_inference_56ed1c34,
  title = {pragmatic reward inference},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/pragmatic-reward-inference/rdr_56ed1c34262c4905bd1c94eff8d15e06},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_pragmatic_reward_inference_56ed1c34_v1,
  title = {pragmatic reward inference},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/pragmatic-reward-inference/rdr_56ed1c34262c4905bd1c94eff8d15e06/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake