People and learned agents who weight the present too heavily often start multi-step tasks and then stall. They procrastinate on the next step or abandon the task partway through. Defaults, meaning options that take effect unless the agent actively opts out, help while they are switched on. The evidence says they stop helping the moment they are removed: in a common-pool resource game, behavior returned to near the selfish equilibrium as soon as defaults were withdrawn (From self-interest to collective action). Control theory for dynamical systems tells a different story. A finite push can have permanent effects if it moves the system across a threshold into a region from which its own dynamics carry it to the desired outcome (Target Control of Asynchronous Boolean Networks, Catastrophe by Design in Population Games).
This proposal reconciles the two views for a specific and common case: present-biased agents doing multi-stage tasks. For such an agent, task progress itself creates the threshold. Near the end of a task, the remaining effort is small enough that even an impatient agent finishes. Early on, it is not. Our proposal is to compute exactly where that threshold lies in a structured task representation. We then design the shortest temporary default schedule that pushes the agent across it, after which the default is switched off and the agent completes the task by itself. The CPRD failure is then explained, not contradicted: that game has no such threshold, so no temporary default could have stuck.
Core Claim
We represent a multi-stage task as a reward machine, a small finite-state automaton whose state records which subtasks have been completed (Using Reward Machines for High-Level Task Specification and Decomposition in Reinforcement Learning). Pairing this automaton state $u$ with the environment state $s$ gives a "progress state" $x=(s,u)$ in which history-dependent rewards become ordinary state-dependent ones.
The agent follows the present-bias model of Time-Inconsistent Planning and Computational Issues in Time-Inconsistent Planning. At progress state $x$, it scores each available action $a$ that leads to $x'$ by its perceived cost $\tilde Q(x,a)=c(x,a)+\beta\,D(x')$. Here $c$ is the immediate effort, $D(x')$ is the true remaining cost to completion, and $\beta\in(0,1]$ is the present-bias weight. Quitting is also an option, with perceived cost $\beta R$ for task reward $R$.
We add one ingredient: inertia. When the designer places a default $d$ at $x$, the agent accepts it whenever
$$\tilde Q(x,d)\;\le\;\min_a \tilde Q(x,a)+\kappa,$$
where $\kappa\ge 0$ is the effort it takes to override a default. Inertia of this kind is strong in practice: most programmers given a cooperative default trading agent left its code untouched (Providing a Recommended Trading Agent to a Population).
The self-completion set $\mathcal C$ is the set of progress states from which the agent, with no defaults, reaches an accepting automaton state. The temporary-default problem is to find a schedule, specifying which progress states get a default and which default each one gets, that drives the agent into $\mathcal C$ while minimizing the number of defaulted steps. The default is withdrawn once the agent is inside $\mathcal C$.
The central technical conjecture concerns why soft defaults should be easier to design than hard restrictions. A default does not delete edges, so it leaves the agent's perceived remaining cost $D$ unchanged. Tang et al. show that computing an optimal "motivating subgraph" by deleting edges is NP-hard. Part of what makes deletion hard is that every deletion changes $D$ everywhere upstream. Under defaults, each state's free choice and its set of acceptable defaults are fixed in advance. Designing the schedule then reduces to a shortest-path problem on the progress graph: free moves cost 0, defaulted moves cost 1, and the target is $\mathcal C$. That problem is solvable in polynomial time, and in the stochastic case it becomes a stochastic shortest-path problem solvable by linear programming. Proving or refuting this reduction, and finding where it breaks, is the first deliverable.
Research Program
Theory for naive agents. The first step is to characterize $\mathcal C$ for naive agents, who assume their future selves will follow today's plan. We then establish the polynomial reduction for the deterministic case. We also bound the gap between the cheapest temporary default schedule and the cost of a permanent default, as a function of $\beta$, $\kappa$ and graph structure. Akerlof-structured graphs are the natural stress case, since Tang et al. show they are worst-case for cost inefficiency.
Theory for harder agents. We extend the analysis to sophisticated agents, who anticipate their own future reversals (Planning Problems for Sophisticated Agents with Present Bias). For these agents, $\mathcal C$ is defined through the equilibrium among the agent's successive selves, and defaults may interact with self-commitment. We expect the clean reduction to fail here and want to locate where. We also treat variable present bias, where $\beta$ fluctuates over time (Procrastination with variable present bias). In that case $\mathcal C$ becomes a probabilistic object: the set of states from which completion happens with at least a target probability.
Robustness to a misjudged agent. The designer rarely knows $\beta$ and $\kappa$ exactly. We therefore compute schedules that remain valid over an interval of these parameters. We also measure how quickly a schedule designed for one $\beta$ degrades when the agent's true $\beta$ differs. This matters because the inverse-RL literature warns that discounting parameters are only partially identifiable from behavior (Partial Identifiability in Inverse Reinforcement Learning for Agents with Non-Exponential Discounting).
Connection to multi-timescale objectives. We also connect the work to Pitis's Procrastinator's Peril (Consistent Aggregation of Objectives with Diverse Time Preferences Requires Non-Markovian Rewards). Pitis fixes the agent's objective by adding history-dependent weights. We leave the agent's objective untouched and instead intervene from outside, briefly. This is the relevant regime whenever the designer cannot rewrite the agent's preferences.
Baselines
Do nothing. The naive reference is the agent with no intervention at all. It yields the completion rate and realized cost of an unassisted present-biased agent. In the Procrastinator's Peril, the unassisted aggregator plans a trajectory valued at 3.20 but realizes 1.00, the lowest attainable value (Pitis).
Permanent default. The always-on default places the optimal action as the default at every state for the whole episode. This is how defaults are used in practice, from automatic enrollment to pre-filled electronic health record orders (BE IMMUNE trial design). It is the strongest competitor on completion rate, because it cannot do worse than our schedule. The comparison is therefore about intervention cost. Beating it means matching its completion rate within statistical noise while using strictly fewer defaulted steps, and our target is at least a 2x reduction on Akerlof-type tasks. Both methods face the same agents, the same $\beta$ and $\kappa$, and the same task graphs.
Intermediate rewards. This baseline applies the polynomial $2n$-approximation for placing intermediate rewards from Computational Issues in Time-Inconsistent Planning, along with exact edge deletion on small graphs. These are the strongest existing methods for steering present-biased planners. No reported empirical result was found on Lacuna for either method, since the paper's results are theoretical, so we will implement both ourselves. To make the comparison fair, we convert defaults to a common currency: a defaulted step is charged $\kappa$, the agent's override cost, which is what the default is worth to the agent. Beating these baselines means reaching equal completion at lower total intervention cost, with lower computation time on large graphs.
Front-loaded default. This is a naive temporary schedule that defaults the first $k$ steps, with $k$ tuned per task family on held-out instances using the same tuning budget we use. It is the ablation that isolates our key ingredient, the computed self-completion set.
Step-counter switch-off. A second ablation computes the threshold in elapsed time rather than in automaton state. It tests whether representing task progress as an automaton state is necessary.
Why not just…?
Why not just keep the default on permanently? It works, and our experiments will report exactly that. The question is what it costs. Permanent interventions carry ongoing expense and fragility (A Dynamics-based Approach for the Target Control of Boolean Networks). In settings with human users, a lasting default also raises autonomy concerns, and a lasting harmful default anchors behavior strongly (From self-interest to collective action). If the temporary schedule saves few steps, for example because $\mathcal C$ contains only the last state, then the permanent default is the right answer, and we will report that as a finding about which task structures admit persistence.
Why not just default the first few steps? This is the most serious threat. If the self-completion set happens to be an upper tail of a linear task, a tuned front-loaded window could match our schedule. We expect it to fail in branching or looping tasks, where the cheapest route into $\mathcal C$ goes through middle stages or alternates between free and defaulted moves. We also expect it to fail under shifts in $\beta$, because a tuned $k$ encodes no structure. The front-loaded ablation on Office World and Minecraft-like tasks with branching automata will settle the question. If it comes within 10% of our schedule's cost across task families, we will narrow the claim to computing $\mathcal C$, which remains useful as a certificate that a given window suffices.
Why not just add a step counter to the state instead of using a reward machine? Elapsed time does not identify progress: two agents at step 20 may have finished different subtasks. Toro Icarte et al. also report that RM-based learners find globally optimal routes where hierarchical baselines get stuck (Reward Machines). The step-counter ablation tests this directly.
Why not just shape rewards and train PPO? Optimal reward placement for present-biased agents is NP-hard (Tang et al.). Moreover, Bellman-based learners such as PPO are biased under non-exponential discounting (Beyond the Bellman Recursion). We include the reward-placement baseline above, but we do not expect shaping to offer a cheaper route.
Proposed Experiments and Decisive Checks
Environments. Simulated naive, sophisticated and variable-bias agents act on four kinds of tasks. The first two are Akerlof graphs and random acyclic graphs from the planning literature. The third is the Procrastinator's Peril. The fourth is reward-machine tasks in Office World and the Minecraft-like world (Reward Machines); no numeric results for these were found on Lacuna, so they serve as task structures, not as score comparisons. Button World's cyclic and repeated-completion tasks (Aronis et al.) cover looping progress.
Parameters and metrics. We sweep $\beta$ from 0.3 to 0.95 and $\kappa$ from 0 to the median edge cost. Metrics are completion rate after the default is withdrawn, the number of defaulted steps, total intervention cost, realized agent cost relative to the optimum, and solver runtime.
Check 1: the threshold is causal. We randomize the step at which the default is withdrawn. Withdrawing inside $\mathcal C$ should yield near-100% completion. Withdrawing one state before entering $\mathcal C$ should yield the unassisted rate. A smooth dose-response curve instead of a step would falsify the threshold picture.
Check 2: scaling. The shortest-path solver should scale polynomially on graphs where exact edge deletion becomes intractable. We test this up to thousands of progress states.
Check 3: a negative control. We model the CPRD game as repeated play by present-biased agents. For that game our analysis should return an empty $\mathcal C$ for the pro-social state, since its only attractor is near the Nash level of 18 tokens. That would reproduce the observed reversion to 16–17 tokens and the absence of a persisting effect (p = 0.53) (From self-interest to collective action). Finding persistence there would indicate a modeling error.
Check 4: robustness. Schedules designed for a nominal $\beta$ are evaluated on agents whose true $\beta$ is perturbed by ±0.1. Interval-robust schedules should retain completion rates that nominal schedules lose.
References
- From self-interest to collective action: The role of defaults in governing common resources
- Target Control of Asynchronous Boolean Networks
- Catastrophe by Design in Population Games: A Mechanism to Destabilize Inefficient Locked-in Technologies
- Using Reward Machines for High-Level Task Specification and Decomposition in Reinforcement Learning
- Time-Inconsistent Planning: A Computational Problem in Behavioral Economics
- Computational Issues in Time-Inconsistent Planning
- Providing a Recommended Trading Agent to a Population: A Novel Approach
- Planning Problems for Sophisticated Agents with Present Bias
- Procrastination with Variable Present Bias
- Partial Identifiability in Inverse Reinforcement Learning for Agents with Non-Exponential Discounting
- Consistent Aggregation of Objectives with Diverse Time Preferences Requires Non-Markovian Rewards
- Behavioural economics to improve and motivate vaccination in primary care using nudges through the electronic health record: rationale and design of the BE IMMUNE randomised clinical trial
- A Dynamics-based Approach for the Target Control of Boolean Networks
- Beyond the Bellman Recursion: A Pontryagin-Guided Framework for Non-Exponential Discounting
- Leveraging Reward Machines for Efficient Multi-Objective Reinforcement Learning