Separating Reward from Present Bias Using What People Say and What They Do

People often say one thing about their goals and do another. Someone tells a fitness assistant that "consistent exercise matters most to me" and then skips the gym tonight. A system that learns preferences by watching behavior will conclude that this person does not much value exercise. A system that follows language literally will overlook the skipped workouts. Both conclusions are wrong. The human's preferences may be stable while the human is present-biased: immediate costs weigh disproportionately at the moment of acting, so plans made in advance are later abandoned. This note proposes an assistant that treats talk and action as two outputs of one person observed at two different moments. Abstract statements about what matters are produced while planning for the future, where present bias largely cancels out. Choices are produced at the moment of acting, where it does not. By inverting both channels jointly, the assistant can recover the person's reward function and their degree of present bias separately. Neither channel can do this alone.

The reason this matters is a negative result. Partial Identifiability in Inverse Reinforcement Learning for Agents with Non-Exponential Discounting proves that when a demonstrator discounts the future non-exponentially, the reward inferred from their behavior cannot predict how they would act under a different discounting rule. That paper calls this the transfer problem, and it is a roadblock for aligning an assistant with a person's considered goals rather than their impulses. Its analysis uses state-action trajectories only. Our claim is that pragmatically interpreted language supplies exactly the missing constraint, because it is a second observation of the same reward generated under a different effective discount curve.

Core Claim

We model the human with quasi-hyperbolic discounting, a standard model of present bias. An outcome $t$ steps ahead is weighted by $D(0)=1$ and $D(t)=\beta\delta^t$ for $t\ge 1$. Here $\delta\in(0,1)$ is ordinary patience and $\beta\in(0,1]$ is present bias ($\beta=1$ recovers exponential discounting). The fact we exploit is elementary. When a person compares outcomes that all lie in the future, the factor $\beta$ multiplies every term and drops out of the comparison. The person then judges consistently, as an exponential discounter would. When one option is immediate, $\beta$ does not cancel.

We ground this in the rational-speech-acts tradition, which models an utterance as chosen by a speaker who is trying to help a listener act well. Following How to talk so AI will learn: Instructions, descriptions, and autonomy, the speaker chooses utterance $u$ by the expected utility the listener would obtain after hearing it. Our speaker, however, evaluates that utility with the planning self's discount curve, while the human's own choices come from the acting self. For a linear reward $R(s)=\theta^\top\phi(s)$, with $\phi(s)$ the features of state $s$ and $\theta$ their weights, the listener's joint posterior is

$$P(\theta,\beta,\delta,H \mid u_{1:m}, \tau) \propto \prod_{j} P_{\text{speak}}(u_j \mid \theta,\delta,H)\; P_{\text{act}}(\tau \mid \theta,\beta,\delta)\; P(\theta)P(\beta)P(\delta)P(H).$$

In this expression, $u_{1:m}$ are the utterances, $\tau$ is the observed trajectory of choices, $H$ is the autonomy horizon from Sumers et al. (how long the listener will act unsupervised), $P_{\text{speak}}$ is a softmax speaker scoring utterances by listener value under $\delta$ alone, and $P_{\text{act}}$ is a softmax naïve present-biased actor of the kind formalized by Skalse and Abate. The asymmetry is the key choice. Descriptions ("swamps are dangerous", "cost matters more than duration") are evaluations of features in general, and we model them as $\beta$-free. Actions are $\beta$-laden.

The theoretical claim we will prove is this. In finite-horizon problems with linear features, the description channel identifies $\theta$ up to positive scale; it does not leave the potential-shaping ambiguity of trajectories, because descriptions refer to features rather than to values of states. Given $\theta$, the pattern of action-level deviations then identifies $\beta$. Actions alone leave $\theta$ and $\beta$ confounded, which is the content of Skalse and Abate's theorems restricted to this family. Language alone identifies $\theta$ only on the features that get mentioned and says nothing about $\beta$.

A second consequence resolves an ambiguity left open by Sumers et al. Their listener reads an instruction such as "take this path now" as evidence of a short horizon $H$. Under our model, the same instruction could come from a long-horizon speaker whose present bias makes the immediate step salient. These two readings make different predictions. An instruction that agrees with the speaker's own descriptions points to a short $H$. An instruction that contradicts them, favoring the low-effort immediate option, points to low $\beta$. The listener can therefore tell the two apart only if it models both.

Research Program

The work has three parts.

The first part is theory. We will prove the identifiability result above for the quasi-hyperbolic, linear-feature, finite-horizon case. We will characterize when it fails. Instructions, unlike descriptions, carry $\beta$, so a speaker who only issues instructions offers no escape from the confound. A sophisticated speaker, one who anticipates their own future bias, may deliberately issue instructions as a commitment device, echoing the sophistication effects in Planning Problems for Sophisticated Agents with Present Bias. We treat naïve versus sophisticated as a binary latent variable rather than assuming one. Kleinberg et al. show that underestimating bias produces exponential failures while overestimating it is safe, so we will also report what the posterior over $\beta$ implies for choosing interventions conservatively.

The second part is the inference machinery. The speaker model extends the combined action-and-reward speaker of Inferring Rewards from Language in Context, which trades off helping with the current choice against describing general preferences. Our extension gives the reward-description term the $\beta$-free planning curve. We adopt the Gricean omission rule of Pragmatic Feature Preferences, under which unmentioned features are treated as irrelevant, and use its augmentation so that language covers more features. The actor model reuses the Boltzmann-naïve and Boltzmann-sophisticated policies of Skalse and Abate, computed by backward induction over finite horizons. Utterances are mapped to feature statements with the same kind of language-model parser used in the Prag FP study. We are not trying to solve lexicon grounding here.

The third part is a new task, because no benchmark in the file pairs utterances with present-biased action. We will build a multi-stage version of the mushroom-foraging and flight-booking environments. In it, options carry immediate effort costs and delayed rewards, human participants describe their preferences to an assistant before a sequence of choices, and they then make those choices under real delays. Real-effort tasks spread across two sessions a few days apart supply those delays. This directly targets the step from single-stage choice problems to multi-stage sequential problems, which the file identifies as open for pragmatic inference.

Baselines

The naive reference point is action-only Boltzmann inverse reinforcement learning with exponential discounting. It treats every choice as a noisy optimal choice under a fitted $\gamma$ and ignores language. It is the default preference learner, and every other method must beat it.

The strongest existing language method is the latent-horizon pragmatic listener of How to talk so AI will learn. We will give it the same trajectories through a standard exponential action likelihood, so that it sees exactly the data our model sees. Its only missing ingredient is present bias. On the one benchmark with reported numbers, FLIGHTPREF, the closely related action-plus-reward listener of Lin et al. reports 59.1% held-out accuracy, against 57.8% for literal language and 52.8% for action-only (Inferring Rewards from Language in Context). For the mushroom-foraging listener, the file contains no numeric figure: no reported result found on Lacuna for the latent-horizon listener's reward-recovery accuracy. Because FLIGHTPREF choices are one-shot and involve no delays, $\beta$ should be unidentified there. The correct result for our model on FLIGHTPREF is therefore a tie within noise at about 59.1%. Clearly falling below that would mean the added parameters are hurting.

The strongest existing present-bias method is action-only Boltzmann-naïve and Boltzmann-sophisticated inference from Partial Identifiability in Inverse Reinforcement Learning for Agents with Non-Exponential Discounting, with $\beta$ as a latent variable. It isolates what language contributes.

The key-ingredient ablation is our full model with the channel asymmetry removed, so that utterances are also generated under $\beta$. This tests whether the gain comes from the planning-self versus acting-self distinction rather than simply from having more parameters.

All methods will share the same parser output, the same priors over $\theta$, the same inference method (discretized posterior grids for low-dimensional $\theta$), and the same hyperparameter search budget selected on a validation split. We will count our model as beating a baseline if it shows a statistically significant improvement, by paired bootstrap over participants, on the transfer metric defined below.

Why not just…?

Why not just fit a small exponential discount factor and add Boltzmann noise? A single $\gamma$ is uniformly myopic and cannot produce a preference reversal, a point that follows from the uniqueness result in Time Consistent Discounting. Optimal and sub-optimal temporal decisions can explain procrastination in a real-world task found that exponential models failed to reproduce deadline cramming, which only a time-inconsistent model captured. Noise is symmetric, whereas present bias pushes choices in one direction, toward whichever option is immediately easy. Our transfer test settles this empirically. It asks each model to predict choices after the delay structure changes, for example when a far-future choice becomes immediate. That is precisely where an exponential-plus-noise fit should fail.

Why not just trust the language and ignore behavior? This is a real threat to the reward-recovery claim. If people describe every relevant feature, a language-only pragmatic listener may recover $\theta$ as well as our model. We therefore do not rest the contribution on $\theta$ alone. Language-only inference cannot predict when this person will defect from their own plan, because it contains no $\beta$. Choosing a helpful intervention, such as a sub-deadline or removing a tempting option, depends on $\beta$ and on sophistication. We will also report reward recovery broken down by features the person did and did not mention, since actions should add information mainly on the unmentioned ones. If language-only ties us on both reward recovery and unmentioned features, we will report that honestly. The contribution would then narrow to behavior prediction and assistance.

Why not just learn two separate reward functions, a "stated" one and a "revealed" one, and flag disagreements? This alternative describes the gap without explaining it. It does not tie disagreement to immediacy, so it cannot predict which future choices will reverse. Our model makes a falsifiable prediction the two-reward model does not. Disagreement should concentrate on choices with immediate costs and vanish when all options are delayed. We will test that directly.

Proposed Experiments

In simulation, we sample ground-truth $(\theta,\beta,\delta,H)$ and naïve or sophisticated types. We generate utterances from the speaker model and trajectories on multi-stage foraging tasks and on the directed-graph task constructions of Kleinberg et al. The decisive check is that the posterior over $\beta$ concentrates only when both channels are present. With actions only, the posterior should show a ridge along which $\theta$ and $\beta$ trade off, matching the theory. A secondary check stress-tests the model by generating speakers whose descriptions are mildly present-biased, to measure how gracefully identification degrades when the channel asymmetry is only approximate.

With humans, the primary metric is transfer accuracy. We train on session one, then predict session-two choices, including items whose delays have shifted so that previously distant options are now immediate. Secondary metrics are held-out reward recovery (accuracy on choices between purely delayed options, which reflects $\theta$ free of $\beta$) and likelihood of the person's immediate choices. We will preregister the prediction that the gap between each participant's descriptions and their choices predicts their reversals on held-out items. We will also preregister that the inferred $\beta$ correlates with that participant's procrastination on the session-two effort task.

The semester pacing dataset used by Chebolu and Dayan contains actions but no language. We will use it only as a check that our actor model fits real deadline behavior. As an assistance test, in simulated agents and graph tasks we pick commitment interventions by the posterior over $\beta$ and sophistication and compare the realized cost ratio against interventions chosen by the exponential baseline. Here we report results against the bound $b$ that Kleinberg et al. prove for sophisticated agents.

The project fails cleanly if $\beta$ is no better identified with language than without it in the human data, or if the asymmetric model does no better than the symmetric ablation on transfer.