Jittering Time, Not Reward: Exploration for Timing Decisions

Many decisions are really decisions about when: how long to wait before responding, when to give up on gathering evidence, when to act before a deadline. Animals make these choices with a noisy internal clock whose error grows in proportion to the interval being timed. This is called scalar variability, or Weber's law for time. Under that noise, humans and rodents still set their target waiting times close to optimally. A separate line of reinforcement learning work explores by adding random noise to the rewards an agent has already seen, then acting greedily on the noisy estimates. This "perturbed history" approach is cheap, but it treats every action as an unrelated slot and adds noise to reward magnitudes. On timing problems that is the wrong place for the noise. Neighboring durations are nearly the same action, and the payoff often has a cliff: a response a little too early earns nothing and resets the trial.

We propose to put the randomness on the time axis. Each time the agent decides, every past wait time in its history is shifted by multiplicative noise in log-time. The shift has two parts: a fixed width matched to the agent's own timing noise, and an extra part that shrinks as a duration region is sampled more often. The agent refits its reward-rate estimate on this shifted history and acts greedily. The central prediction is that this one change does three things. It pools evidence across neighboring durations at the right scale. It explores without pushing the agent over the payoff cliff, which symmetric reward noise does. And without being told the payoff structure, it pushes the agent's aim point past the deadline by an amount proportional to its timing noise, which is the normative bias Balcı and colleagues derived.

Core Claim

Optimal Temporal Risk Assessment supplies the target behavior. When early responses earn nothing, the reward-maximizing aim point sits past the objective deadline by an amount proportional to the timing coefficient of variation. That paper also reports that humans and rats reach 96–99% of maximum expected gain. It explains what the optimum is, but not how an agent that does not know the payoff curve should learn it.

The perturbation literature supplies the learning mechanism:

All three perturb reward values independently per action or per step. None uses the geometry of time.

Our perturbation acts on the action coordinate instead. Each past wait $w_i$ is replaced by

$$\tilde w_i = w_i \exp(\sigma_i \xi_i), \qquad \sigma_i^2 = \hat\kappa^2 + \frac{\beta^2}{n(w_i)},$$

where:

  • $\xi_i$ is a fresh standard normal draw at every decision;
  • $\hat\kappa$ is the agent's estimate of its own timing coefficient of variation, obtained from its repeated productions;
  • $n(w_i)$ is the number of past waits in a log-time window around $w_i$;
  • $\beta$ sets the epistemic exploration scale.

The agent then fits a reward-rate curve on the shifted history using local averaging on a log-time grid, adds a small PHE-style block of pseudo-rewards to keep unvisited durations optimistic, and picks the duration that maximizes the fitted curve.

The $\hat\kappa$ term is not exploration. It encodes the agent's prior belief that intended durations closer together than the agent's own timing precision earn nearly the same expected reward. That is approximately true because each intended wait already lands on the true payoff curve blurred by the agent's own timing noise. The $\beta$ term carries the exploration and shrinks with data, roughly as a standard error does.

What is new here:

  1. Perturbing the time-stamps of history rather than its rewards. This is an untested design. The research file found no paper that unifies endogenous timing noise with perturbation-based exploration.
  2. The claim that it handles cliff-shaped temporal payoffs better than symmetric reward noise.
  3. The claim that it recovers the Balcı aim-point shift as an emergent property.

Research Program

Stage one: timing bandits. We build simulated versions of the three task families that Balcı and colleagues analyzed.

  • Differential reinforcement of low rates (DRL): a response before a minimum wait earns nothing and resets the trial.
  • Switch task: the agent must choose when to switch between two locations at which reward is scheduled at different times.
  • Beat-the-clock: reward grows with the wait but is lost if a deadline passes.

Each simulated agent chooses an intended duration on a log-spaced grid of about 50 to 100 values. The executed duration is the intended one multiplied by $(1+\kappa\epsilon)$, with $\kappa \in \{0.1, 0.2, 0.3\}$ and $\epsilon$ standard normal. We test two observation conditions. In the first, the agent sees only its intended duration and the reward, which is the subjective-time case. In the second, it also sees a noisy reading of the executed duration.

We also add a "when to stop" variant with a stochastic deadline. Following Sequential Hypothesis Testing under Stochastic Deadlines, its optimal stopping rule loosens the evidence threshold as time runs out. This variant asks whether time-jittered exploration learns that collapsing rule when the deadline distribution is unknown.

Stage two: theory. Russo's sufficient-optimism argument requires a constant probability that the perturbed estimate overestimates the value of the optimal action. Shifting time alone cannot guarantee this once data are dense, because the shifts average out. This is why the $\beta/\sqrt{n}$ term and the pseudo-rewards are needed. We will try to prove a gap-dependent regret bound for the combined scheme on Lipschitz-in-log-time reward-rate curves. We will also test whether a threshold appears in $\beta$ that mirrors PHE's empirical behavior: PHE matches Thompson sampling at $a=1.1$ and suffers linear regret at $a=0.5$, while its theory requires $a>2$.

Stage three: deep agents. We port the idea to deep RL on the PsychLab interval reproduction task used in Interval timing in deep reinforcement learning agents. There, recurrent agents built ramping internal timers, while feedforward agents built brittle "behavioral clocks" out of stereotyped movement, a strategy the authors call autostigmergy. The deep analogue of our scheme does not add noise to the reward value. Instead it moves the step at which reward is credited by a log-normal shift whose width matches the scalar noise.

We ask two questions. Does this temporal credit jitter favor internal timers that generalize, over movement-based clocks? Does reward-magnitude noise in the style of Exploration by Random Reward Perturbation do the same, or the opposite? To read out timer formation, we use the same principal-component ramping analysis as Deverett et al.

Baselines

Naive reference. An ε-greedy agent over the duration grid, tuned over five values of ε. It shows how much any structured exploration buys.

Strongest classical randomized method. Per-bin Beta-Bernoulli Thompson sampling on reward rate, together with PHE at $a=1.1$ as specified in Perturbed-History Exploration in Stochastic Multi-Armed Bandits. These are the strongest competitors because they define the regret frontier for independent-arm randomized exploration. The file reports only a qualitative result for them: PHE at $a=1.1$ performs nearly identically to Thompson sampling and beats UCB1. I found no reported regret number on Lacuna for either method on timing tasks.

Reward-magnitude perturbation. Gaussian reward noise with linear annealing, following Exploration by Random Reward Perturbation, with its noise scale tuned on the same grid budget as our $\beta$. It is the direct test of "perturb reward" against "perturb time."

Plug-in oracle. An oracle that knows the payoff form and the true $\kappa$ and plays the expected-gain-maximizing aim point of Optimal Temporal Risk Assessment. It anchors the scale of performance. The only numbers available are behavioral: humans reach 98% of maximum expected gain on DRL and on the switch task, rats reach 96–98% on DRL, and mice reach 99% on the switch task. We will report our agents' fraction of maximum expected gain on the same tasks so the comparison with biology is direct. Beating the learning baselines means reaching a given fraction of maximum expected gain, for example 96%, in significantly fewer trials across 50 seeds. It also means a lower rate of premature responses than reward-noise exploration.

Ablations. Three ablations isolate the key ingredient:

  • Correlated reward noise: Gaussian reward noise smoothed with the same log-time kernel as our shifts. This separates "perturb time" from "share information across neighbors."
  • Linear-time shifts: shifts of the same average size applied in linear rather than log time. This tests whether the Weber shape matters.
  • Mismatched $\hat\kappa$: setting $\hat\kappa$ to half or double the true $\kappa$. This tests whether matching the agent's own noise matters.

Fairness controls. All methods share the same grid, trial budget, seeds, and a five-point tuning sweep for their single exploration parameter.

Why not just…?

Why not just use ε-greedy or an entropy bonus? Russo argues that dithering lacks deep exploration. Efficient Exploration via Epistemic-Risk-Seeking Policy Optimization reports that a vanilla actor-critic fails on DeepSea around depth 14. On timing bandits, though, the horizon is short, so dithering's main cost should instead be premature responses that trigger resets. The ε-greedy baseline measures that cost directly. If it matches our method on DRL, the claim reduces to the deep-agent stage.

Why not just fit a Gaussian process over log-time and run Thompson sampling on it? This is the real threat in stage one. A Gaussian process (a smooth function prior) with a log-time kernel of width $\kappa$ shares information across durations in a principled way, and it might match our method. The file contains no reported result on Lacuna for Gaussian-process bandits on timing tasks, so we will run it as an additional baseline.

If it ties, we will say so. The remaining case for our scheme would then be the same one Learning Contextual Bandits Through Perturbed Rewards makes for perturbation over posterior methods: it avoids costly matrix inversions and carries over to neural agents (stage three), where an exact posterior is unavailable.

Why not just tune the reward-noise scale, or add elapsed time to the state? Symmetric reward noise has no notion of which durations are close to each other. On a cliff-shaped payoff it should push the greedy choice below the threshold as often as above it, so the prediction is a measurably higher premature-response rate at matched regret. Adding time to the state does not remove noise in the clock itself. Deverett et al. show that agents relying on external time cues build brittle movement-based clocks. The tuned reward-noise baseline and the correlated-reward-noise ablation settle this question.

Why not just estimate $\kappa$ and compute the optimal aim point directly? That works when the payoff form is known, and our oracle baseline does exactly this. Our method targets the setting where the payoff curve must be learned. The test is whether it approaches the oracle without being given the payoff form.

Decisive Checks

Four results would decide the proposal.

  1. Aim-point shift. On DRL with unknown payoff, the learned aim point should move past the threshold in proportion to $\kappa$, matching the Balcı prediction. The reward-noise and Thompson baselines, which have no notion of timing noise, should show weaker or noisier dependence on $\kappa$.
  2. Premature responses. At matched cumulative regret, our method should produce fewer premature responses than the reward-noise baseline.
  3. The Weber shape matters. The linear-time and mismatched-$\hat\kappa$ ablations should degrade performance. If they do not, the log-time framing is decorative.
  4. Deep agents. In PsychLab, temporal credit jitter should increase the share of seeds that develop ramping internal timers and should improve generalization to held-out intervals relative to reward-magnitude noise.

Failure of the first or third check would show that time-stamp perturbation is just smoothing. Failure of the fourth would confine the contribution to timing bandits.