Present Bias Survives the Average-Reward Limit: Identifying and Correcting It in Continuing-Task Reward Learning

Average-reward inverse reinforcement learning infers what an agent wants by watching it act in a task that never ends, and it scores behavior by long-run reward per step. Its main selling point is that it drops the discount factor, the "patience" parameter whose slight misestimation is known to wreck reward inference. Humans, however, are not just impatient by a constant factor. They are present-biased: the next moment counts for much more than anything after it, while the distant future is weighted fairly evenly. We show that this kind of impatience does not wash out when one passes to the long-run average. In the limit where the demonstrator's long-range discount approaches one, an exponential discount factor disappears from the choice rule, but the present-bias factor $\beta$ remains as a multiplier on the agent's own future value. An observer running standard average-reward IRL will therefore fold $\beta$ into the reward, and that distortion will be exposed as soon as the dynamics change.

The proposal has three parts. First, we derive the two behavior models a present-biased agent obeys in a continuing task, one for naïve agents and one for sophisticated agents. Second, we characterize the distorted reward that a misspecified average-reward observer recovers. Third, we test a conjecture: $\beta$ cannot be identified from one environment, but it can be identified from demonstrations under two or more transition dynamics that share a reward. If the conjecture holds, an average-reward IRL method that fits $\beta$ jointly with the reward should transfer to new dynamics in cases where both standard average-reward IRL and discount-fitted IRL fail.

Core Claim

Quasi-hyperbolic discounting is the standard model of present bias. It weights the current step by 1 and step $t \ge 1$ by $\beta\delta^t$, with $\beta<1$. Under this discounting, a choice at state $s$ scores action $a$ as $r(s,a) + \beta\delta\,\mathbb{E}[V_\delta(s')]$, where $V_\delta$ is the continuation value.

As $\delta \to 1$, the Laurent expansion gives $V_\delta = \rho/(1-\delta) + h + O(1-\delta)$, where $\rho$ is the long-run gain and $h$ the relative value (bias) function. In a unichain process the $\rho$ term is the same for every action, so it cancels out of any softmax. Writing $\lambda$ for the Boltzmann rationality temperature, the limiting behavior models are

$$\pi_{\text{naive}}(a\mid s) \propto \exp\Big(\lambda\big[r(s,a) + \beta \textstyle\sum_{s'} P(s'\mid s,a)\, h^{*}_r(s')\big]\Big),$$

$$\pi_{\text{soph}}(a\mid s) \propto \exp\Big(\lambda\big[r(s,a) + \beta \textstyle\sum_{s'} P(s'\mid s,a)\, h^{\pi_{\text{soph}}}_r(s')\big]\Big).$$

In these models, $h^{*}_r$ is the optimal bias under the true reward $r$. The naïve agent assumes its future selves will act optimally. In the sophisticated model, $h^{\pi}_r$ is the bias of the agent's own equilibrium policy, so that model is a fixed point. With $\beta = 1$, both reduce to the Boltzmann-over-average-reward-advantage model assumed by Inverse Reinforcement Learning with the Average Reward Criterion.

The new claims are the following. First, removing $\gamma$ does not remove $\beta$. Second, the reward an average-reward observer recovers is $r$ shifted by a term of order $(1-\beta)\,P h$, and this shift depends on the dynamics $P$. That dependence is why it breaks transfer, and also why it is, in principle, identifiable across environments.

Relation to Prior Work

Several papers motivate the problem by documenting fragility. Quantifying the Sensitivity of Inverse Reinforcement Learning to Misspecification shows that assuming $\gamma=0.99$ when the agent uses $0.98$ can push the inferred reward to a STARC distance of at least 0.5 from the truth. Misspecification in Inverse Reinforcement Learning proves that no standard model is robust to discount misspecification, and that Boltzmann models are robust to temperature errors only when choices stay monotone in value. Both papers stay within geometric families on both sides. We study a non-geometric demonstrator paired with an average-reward observer, which neither paper covers.

Partial Identifiability in Inverse Reinforcement Learning for Agents with Non-Exponential Discounting supplies our taxonomy of naïve, sophisticated, and resolute agents. It also supplies the Transfer Problem: rewards learned under one discount profile fail under another. However, it is explicitly limited to episodic settings because non-exponential sums diverge. Our $\delta\to1$ limit is a direct way past that limitation: the divergent part becomes the constant $\rho/(1-\delta)$, which cancels.

Two papers supply methods we reuse. General Discounting versus Average Reward proves that discounted and average-reward limits coincide for monotone discount sequences whenever both exist. That result motivates treating the average-reward criterion as the right continuing-task normalization. Quasi-hyperbolic sequences are monotone, but their limiting choice rule keeps $\beta$. The asymptotic equality applies to the value of a fixed reward stream, not to the comparison between actions made at each step. Reinforcement Learning with Quasi-Hyperbolic Discounting gives a model-free two-timescale method for finding sophisticated equilibria, and we use it as the forward solver inside our inverse loop. That paper is forward-only and assumes $\beta$ is known. It also warns that equilibria are not unique, which constrains our sophisticated-agent estimator, as discussed below.

Inverse Reinforcement Learning with the Average Reward Criterion supplies the inverse algorithm, Inverse Policy Mirror Descent (IPMD). We extend it by changing only the behavior model. On the Sensitivity of Reward Inference to Misspecified Human Models gives a linear error bound when the behavior model is strongly log-concave in the reward parameters. Because $h^*_r$ is a maximum over policies, our naïve model is not obviously log-concave. Checking whether that bound survives is part of the analysis.

Research Program

The theoretical work has three pieces.

The first piece makes the $\delta\to1$ limit rigorous for unichain MDPs. For naïve agents this means showing that $\pi_{\text{naive}}$ is the limit of finite-$\delta$ naïve behavior. For sophisticated agents it means showing that limits of $\delta$-equilibria are fixed points of the second display equation. We expect the naïve case to be clean and the sophisticated case to inherit multiplicity of equilibria.

The second piece characterizes the misspecified fixed point. Given behavior generated with $\beta<1$, we compute the set of rewards $\hat r$ under which the same policy is Boltzmann-optimal for average reward with $\beta=1$. We then measure, in a STARC-style distance with average-reward canonicalization, how far that set lies from the true reward's shaping class as a function of $1-\beta$ and $\mathrm{sp}(h)$, the span of the bias function. Adapting STARC canonicalization from discounted to average-reward shaping $r + \Phi(s') - \Phi(s)$ is a small but necessary contribution.

The third piece addresses identifiability. In a single environment we expect $(\beta, r)$ to be unidentifiable: any $\beta$ can be paired with some reward that reproduces the policy. We will try to prove this by construction. The main conjecture is that with $K\ge2$ transition kernels that share $r$ and $\beta$ and are generic in a sense to be made precise, $\beta$ is identified and $r$ is identified up to the average-reward shaping class. The intuition is that a true reward change shifts action scores identically across kernels, while the present-bias term $(1-\beta)Ph$ shifts them differently in each kernel.

On the algorithm side, we build β-IPMD. It keeps IPMD's two-timescale structure and replaces the policy step's target with the naïve or sophisticated score from the Core Claim. It also adds a scalar $\beta$ fitted by gradient on the pooled multi-environment likelihood. The naïve variant needs an optimal-bias estimate $h^*_{\hat r}$ from a separate average-reward forward solver. The sophisticated variant uses the quasi-hyperbolic equilibrium learner as its inner loop. Because that learner can converge to different equilibria, the sophisticated variant reports a likelihood maximized over the restarts it reaches, and we report how much the estimates vary across equilibria.

Baselines

The naïve reference point is standard average-reward IPMD fitted on pooled demonstrations from all training environments, with $\beta$ fixed at 1. This is exactly Inverse Reinforcement Learning with the Average Reward Criterion, so this comparison is also the ablation that isolates our key ingredient. The strongest number reported for it is imitation return on Humanoid: 7,379 against an expert score of 7,137. On the same benchmark, IQ-Learn scores 5227.1 according to IQ-Learn: Inverse soft-Q Learning for Imitation. These are in-distribution imitation scores for rational demonstrators. No reported result was found on Lacuna for any IRL method on present-biased demonstrators or on reward transfer across dynamics. All baseline numbers on our metrics will therefore be produced by us.

The second baseline is discounted maximum-entropy IRL with $\gamma$ fitted jointly with the reward on the same data. This is the strongest competitor on our specific question, because a fitted $\gamma<1$ is the obvious way to absorb myopia. The third is IQ-Learn run from the IQ-Learn Codebase for imitation-only comparison.

The ablations are as follows. β-IPMD trained on a single environment tests whether multi-environment data is what makes $\beta$ identifiable. A naïve model is fitted to sophisticated data, and the reverse, to measure how much the agent-type assumption matters. An oracle uses the true $\beta$, which gives the ceiling.

All methods receive identical demonstration sets, network sizes, and environment-step budgets. Each gets the same hyperparameter search budget, and the discount-fitted baseline gets a grid over $\gamma$ of the same size as our $\beta$ search. Beating the baselines means the following. On held-out dynamics, β-IPMD must achieve lower average-reward STARC distance and higher held-out policy log-likelihood than both the IPMD and $\gamma$-fitted baselines. It must also recover $\beta$ within a tolerance we set before running, across at least five seeds. Finally, it must match IPMD within noise when the demonstrator actually has $\beta=1$, so that the extra parameter does no harm.

Why not just…?

Why not just fit IPMD and accept the distorted reward? In the training environment the distorted reward predicts behavior perfectly well, and that is exactly why the problem goes unnoticed. The distortion depends on $P$, so the claim lives or dies on transfer. The decisive check is whether IPMD's held-out-dynamics error grows with $1-\beta$ while β-IPMD's does not. This mirrors the Transfer Problem of Partial Identifiability in Inverse Reinforcement Learning for Agents with Non-Exponential Discounting, moved into continuing tasks.

Why not just fit a discount factor $\gamma<1$ to absorb the myopia? This is the real threat, and we treat it as one. A geometric discount shortens the horizon uniformly. Present bias with $\delta\to1$ instead overweights a single step and then stays fully patient. The two models make different predictions when a kernel change alters long-range bias differences but leaves one-step rewards unchanged. Discount mismatch is also known to be catastrophic for reward inference (Quantifying the Sensitivity of Inverse Reinforcement Learning to Misspecification). If the $\gamma$-fitted baseline matches β-IPMD on transfer for most kernel shifts, then present bias and short horizons are not practically separable. We would report that as a negative result and narrow the claim to the kernel shifts where they diverge.

Why not just raise the Boltzmann temperature? Present bias produces systematic reversals in a fixed direction, not symmetric noise, and temperature robustness requires choices to stay monotone in value (Misspecification in Inverse Reinforcement Learning). The fitted temperature is included in every baseline's tuning, so if noise were sufficient it would show up there.

Why not skip rewards and imitate directly with IQ-Learn? IQ-Learn's implicit reward depends on the training dynamics and horizon. When the kernel changes, that is precisely what we need to transfer. We include IQ-Learn to confirm this on our held-out dynamics. We expect it to do well in-distribution and poorly after a shift.

Proposed Experiments

Tabular continuing MDPs. We start with random unichain MDPs in the style of the STARC suite of STARC: A General Framework For Quantifying Differences Between Reward Functions, which reports a correlation of 0.873 between STARC-VAL distance and regret. We also use a continuing version of the gym-membership preference-reversal MDP. For each true reward we draw $K\in\{1,2,4\}$ kernels and simulate naïve and sophisticated demonstrators with $\beta$ ranging from 0.5 to 1. We then evaluate β-recovery error, reward distance on a held-out kernel, and policy log-likelihood. The first decisive check is a plot of held-out error against $1-\beta$: it should rise for IPMD and stay flat for β-IPMD when $K\ge2$, and stay unidentified when $K=1$.

Stochastic inventory control. This environment comes from Reinforcement Learning with Quasi-Hyperbolic Discounting. It provides sophisticated demonstrators with multiple equilibria, and we vary the demand distributions to obtain different kernels. This stresses whether the inverse estimate stays stable across equilibria.

Continuous control. We use Pendulum-v1 and LunarLanderContinuous-v2, the environments in which IPMD already reports better reward recovery than IQ-Learn. Kernel shifts come from mass, gravity, or wind, following the dynamics shifts used in Robust Inverse Constrained Reinforcement Learning under Model Misspecification. Demonstrators are synthetic naïve agents built from a learned optimal bias. If the human Lunar Lander data of On the Sensitivity of Reward Inference to Misspecified Human Models can be obtained, we fit $\beta$ to it as an exploratory check of whether human control shows measurable present bias. This would be descriptive only, since no ground-truth reward is available.

The project succeeds if, on tabular and continuous benchmarks, β-IPMD with $K\ge2$ recovers $\beta$ and transfers rewards to unseen dynamics where both IPMD and $\gamma$-fitted IRL degrade. It fails informatively in either of two cases: if the $\gamma$-fitted baseline transfers just as well, or if multi-environment data still leaves $\beta$ unidentified.