Information Lifetime: One Drift Rate for Forgetting and Exploring

An agent that learns what a person wants faces two clocks. The first counts down to the end of the session. As fewer turns remain, there is less time to use anything new it learns, so it should explore less. The second clock measures how long a learned preference stays true before the person changes their mind. Bandit theory handles the first clock: a bandit is a problem where an agent repeatedly picks among options with unknown payoffs. Methods for preference drift handle the second clock by down-weighting old feedback. No existing method uses both clocks at once.

This note proposes that both clocks reduce to one quantity, the information lifetime: the expected number of future turns during which a fact learned now will still be both true and usable. The same estimated drift rate should set how fast the agent forgets old feedback and how far ahead it plans when deciding whether to explore. If this is right, an exploration rule tuned for a fixed deadline, such as a finite-horizon index, should be fed the information lifetime instead of the raw number of remaining turns. The drift rate should be estimated online rather than fixed in advance. The test is whether this single coupled parameter matches or beats separately hand-tuned forgetting and exploration schedules. We test it on drifting bandits and on conversational agents that must uncover preferences users did not know they had.

Core Claim

Let $m = T - t$ be the number of turns remaining and $\rho$ the per-turn probability that the user's preference changes. Information acquired at turn $t$ is worth exploiting only on future turns where the session has not ended and no change has occurred since. Its expected usable span is

$$L(m, \rho) = \sum_{k=1}^{m} (1-\rho)^k = \frac{(1-\rho)\left(1-(1-\rho)^m\right)}{\rho}.$$

$L$ approaches $m$ when drift is rare ($\rho m \ll 1$) and saturates at roughly $1/\rho$ when drift is frequent. The matching backward rule weights a feedback sample from turn $t_i$ by $(1-\rho)^{t-t_i}$. That is exactly the exponential discount used by Non-Stationary DPO, which re-weights preference pairs by $\gamma^{T-t_i-1}$, with $\gamma = 1-\rho$.

The proposal is to run a horizon-dependent exploration rule with $L(m,\hat\rho)$ in place of $m$. The estimate $\hat\rho$ comes from an online change-point posterior over the agent's own prediction errors, and the same $\hat\rho$ sets the forgetting weights. Discounted Gittins indices are classical in spirit. What we have not found is a method that couples the forward planning horizon to an estimated drift rate shared with the forgetting rule. We also have not found this coupling applied to agents whose own probes cause preferences to appear.

The rule predicts a specific exploration profile. Far from the deadline, the exploration bonus is flat at a level set by $1/\hat\rho$, not by the full horizon. In the last $\sim 1/\hat\rho$ turns it decays toward greed. After a detected change it jumps back up, because $\hat\rho$ rises and old evidence is discounted. This resolves the tension the research file identifies, where finite-horizon theory says to stop exploring near the end while drift says to keep going. Each argument is right in a different regime, and $L$ says which regime the agent is in.

Relation to Prior Work

Three papers motivate the forward half and supply its machinery.

  • Dimitrakakis derives exploration thresholds as explicit functions of the effective horizon. He approximates the needed posterior with an ensemble of bootstrapped value estimates, which we reuse when exact posteriors are unavailable.
  • Lattimore analyzes the finite-horizon Gittins index, whose bonus grows like $\log^{3/2}(m)$ in the remaining horizon. It is our forward rule with $m$ replaced by $L$.
  • Bayes-OCP shows optimism should decay as a deadline nears.

All three assume the underlying payoffs are fixed. The backward half has two sources.

  • NS-DPO shows that ignoring drift collapses reward accuracy to near chance. It uses a fixed discount and never chooses what data to collect.
  • DEX3.S is the closest existing algorithm. It learns from pairwise preference comparisons under drift and achieves dynamic regret $\tilde O(V_T^{1/3}K^{1/3}T^{2/3})$. Its exploratory mixing rate is constant over time and tuned with knowledge of $T$ and the total drift budget $V_T$. We differ by making that rate a function of $L(m,\hat\rho)$, with $\hat\rho$ estimated online.

Two papers set the application regime.

  • AcCoRD documents "triggered preferences," which a user forms only after the agent shows them an option. Frontier models score as low as 0.09 on these. Its uncertainty-prompting agent, UncReAct, over-asks and exhausts the turn budget. That failure looks to us like exploration with no notion of remaining horizon.
  • WebShop shows agents are weakest at option selection: 38.9 versus 73.9 for humans.

One result appears to cut against us. Ads that Stick proves that intrusive actions should be pushed toward both session endpoints when annoyance decays. A naive transfer to clarifying questions would predict a burst of questions near the end. Our model predicts the opposite for elicitation: late questions have almost no information lifetime. The difference is that an ad's payoff is consumed when it is shown, while an elicitation's payoff arrives on later turns. We treat this as a falsifiable contrast between the two models.

Finally, Wilson et al. find human directed exploration rises from horizon 1 to 6 and then plateaus. That ceiling is what $L$ predicts for anyone who believes the environment drifts, with the plateau near $1/\rho$.

Research Program

The first component is algorithmic. We define the coupled index policy for Gaussian and Bernoulli bandits. It keeps a run-length posterior for change-point detection that yields $\hat\rho$, a discounted posterior over arm means, and an index computed at effective horizon $L(m,\hat\rho)$. We then build a dueling variant. It keeps DEX3.S's structure of two EXP3 learners playing each other, but sets the mixing weight at each round to scale with $L(m,\hat\rho)/m$ rather than holding it fixed.

On the analysis side, we aim for a dynamic-regret bound for the coupled index that recovers Lattimore's rate when $\rho = 0$. We would also show the bound is no worse than DEX3.S's when $\rho$ is known. A full anytime proof under unknown $\rho$ may be out of reach; the empirical program does not depend on it.

The second component lifts the rule into an LLM agent on AcCoRD. A small external controller decides each turn whether to probe or to act on current beliefs. Probing means surfacing a deliberately diverse set of options or asking a question. The value of a probe is estimated as

(probability that an undiscovered or changed preference remains) × (gain from acting on it) × $L(m,\hat\rho)$,

and it is compared against a per-probe friction cost. The probability of an undiscovered preference is updated from how often recent probes elicited new constraints, which is the agent's own discovery rate. The LLM writes the probe or action content; the controller only decides when. This keeps the contribution separable from prompt engineering.

Baselines

Naive reference points. For bandits, these are greedy and ε-greedy with linear decay to zero at $T$. For agents, it is the base ReAct-style agent with no elicitation policy. Beating them is necessary but uninformative.

Horizon-only. This is Lattimore's finite-horizon Gittins index with remaining horizon $m$ and no forgetting. It isolates what drift-awareness adds. We run it on identical arm draws and seeds.

Drift-only. On scalar-reward bandits, this is discounted Thompson sampling, where the discount is applied to posterior counts and no deadline is used. On preference-feedback bandits, it is DEX3.S with its prescribed fixed mixing rate. Both receive oracle-tuned discount or mixing parameters chosen by grid search over the true drift budget. This deliberately hands them the information our method must estimate. DEX3.S is the strongest existing method for drifting preference comparisons. The file reports only its asymptotic regret rates and that it "significantly outperform[s] REX3 and uniform random baselines" on synthetic random-walk and Dirichlet environments. No reported regret value was found on Lacuna for these synthetic settings, so we will reimplement it and report both methods on the same environments.

Decoupled, separately tuned. This is the key ablation. It uses a forgetting discount and an exploration horizon as two independent hyperparameters, grid-searched with an equal tuning budget. If coupling through a single $\hat\rho$ is merely a convenient reparameterization, this baseline will match us. Our claim is that coupled online estimation matches the oracle-tuned decoupled version and beats the decoupled version tuned on a held-out drift rate.

Agent baselines on AcCoRD. All use the same base model and the same turn budget.

  • UncReAct.
  • The same agent with the remaining turn count written into every prompt and an instruction to explore early.
  • A tuned explore-then-commit schedule that probes in the first $k$ turns.
  • A bifurcated schedule following Ads that Stick, which probes at both endpoints.

The reference numbers from AcCoRD are that Claude-4.5 reaches a Perfect Outcome Rate under 30% in both domains, and that Triggered Preference scores fall as low as 0.09 for some models. We will reproduce those baselines with our base model rather than compare across models. Beating them means a higher Triggered Preference score and Perfect Outcome Rate at an equal or smaller number of turns used.

Why not just…?

Why not just tell the LLM how many turns remain? This is the most serious threat on the agent side. A capable model may already reason that exploring late is pointless. It cannot, however, estimate a drift or discovery rate from its own history in a calibrated way. AcCoRD's over-asking failures suggest models do not budget questions well even when they could. We test this directly with the turn-count-in-prompt baseline. If it matches the controller on Triggered Preference score and turns used, the agent-side claim fails, and we would report the bandit results alone.

Why not just use discounted Thompson sampling and stop exploring in the last few turns? Bolting a fixed cutoff onto a fixed discount is the decoupled baseline in disguise. With both knobs tuned on the true drift rate, it may well match us. That is why the evaluation includes a mismatched-tuning condition: tune on one drift rate, test on another, and also on environments where the drift rate itself changes mid-session. A single online $\hat\rho$ should adapt in those conditions while fixed knobs cannot. If the tuned fixed pair is robust across the drift range, the coupling buys only reduced tuning, and we will say so.

Why not just probe on a fixed early schedule? Explore-then-commit is cheap and fits a single deadline. It cannot respond to a preference that changes or is triggered mid-session. Ads that Stick also shows that uniform spacing of intrusive actions is suboptimal by at least 10% when users' annoyance carries over between actions. Our experiments include late-drift scenarios specifically, where any fixed early schedule should fail.

Why not just prompt for uncertainty, as UncReAct does? AcCoRD reports that this did not significantly help and led weaker models to exhaust their budgets. It remains a baseline, but the evidence already argues against it.

Proposed Experiments

Synthetic bandits. We extend the testbeds in the file with controlled drift:

  • Dimitrakakis's 16-, 128- and 256-arm Bernoulli bandits.
  • Lattimore's Gaussian bandits with $n = 10^4$ and 2, 5 or 10 arms.
  • DEX3.S's random-walk and Dirichlet preference-matrix environments.

Drift takes two forms: abrupt switches at a rate $\rho$, and continuous random walks. We sweep $\rho T$ from about 0.1 to about 100. Metrics are dynamic regret, regret in the final 10% of the horizon, and turns needed to recover after a switch.

The decisive check is a phase diagram over $(\rho, T)$. When $\rho T$ is small, the coupled index should coincide with horizon-only Gittins. When $\rho T$ is large, it should coincide with drift-only methods except near the deadline. Its advantage should concentrate where $1/\rho$ is comparable to $T$. If no region of the diagram shows a gain over the mismatched-tuned decoupled baseline, the core idea is wrong.

Agent experiment. We run on AcCoRD-Shop and AcCoRD-Travel, stratified by preference type. Turn budgets vary so the information lifetime actually changes. We add an injected-drift condition in which the simulated user reverses a stated constraint mid-dialogue. Metrics are:

  • Triggered Preference score.
  • Perfect Outcome Rate.
  • Task Completion Rate.
  • Number of probes and their timing.

There are two ablations. Setting $\hat\rho = 0$ gives a horizon-only controller; setting $L = 1/\hat\rho$ removes the deadline. A decisive behavioral check is that probe timing should decay near the deadline and spike after injected drift. A rise in probes near the end, which the naive bifurcation transfer predicts, would contradict our model.

Cognitive prediction (secondary). We would replicate the two-armed horizon task of Wilson et al. while telling participants the payoffs may drift at one of two rates. The information-lifetime account predicts that the increase in the information bonus from horizon 1 to horizon 6 shrinks under high drift, and that the ceiling moves to shorter horizons. Horizon-only accounts predict no effect of announced drift. This is a cheap way to ask whether people already plan with information lifetime rather than raw horizon.