Spending Loss Aversion Wisely: Frame-Aware Actor-Critic Policies for Mobile Health Nudges

Telling people they will lose something works better than telling them they will gain the same thing. On a solar retailer's website, "save €813 a year" beat "earn €813 a year" by about a quarter, and a 30-day trial that gave people money up front and took it back on missed days kept far more of them logging meals when the deductions were unpredictable. A loss frame, though, is probably a resource that wears out: the same people who respond strongly to the first "you're about to lose it" message tend to tune out the twentieth. Today's mobile health systems that decide when to nudge someone, called just-in-time adaptive interventions, learn only whether to send a generic prompt. Behavioral-economics trials, for their part, fix the frame for each person at enrollment. Nobody learns when a loss frame is worth spending and when a gentler gain frame or silence should be used instead.

We propose to treat message framing as part of what a mobile health agent decides, and to learn that decision with an actor-critic method. This is reinforcement learning in which one model, the actor, picks actions and a second model, the critic, estimates how good the user's current situation is for the future. The central bet is that each frame has its own fatigue, and that the money already held in a user's up-front balance is part of their state. Under those conditions, the best action at a given moment depends on what it costs later. A short-sighted learner would keep picking the frame with the biggest immediate effect, which is likely the loss frame, until that frame stops working. A critic that tracks frame-specific fatigue and the remaining balance can learn to ration it. The proposal is built so that this bet can fail cleanly. The first phase measures whether frame-specific fatigue exists at all. If it does not, a simpler method is enough, and we say so.

Core Claim

The claim is that frame-specific fatigue creates delayed costs that a myopic policy cannot see. We model fatigue for each frame $f$ (none, gain, loss-reminder, endowment-clawback) with its own decaying exposure count:

$$X^{(f)}_{t+1} = \lambda_f X^{(f)}_t + \mathbf{1}\{a_t = f\}.$$

Here $a_t$ is the action at decision point $t$, and $\lambda_f \in (0,1)$ is how quickly the effect of past exposures to frame $f$ fades. This extends the single, frame-blind "dosage" variable of Personalized HeartSteps. The critic estimates $V(s)$, the expected future step count, from a state $s$ that combines sensor context, every $X^{(f)}$, and the user's remaining endowed balance $B_t$. The actor outputs two things. The first is a choice among frames, with probabilities kept between 0.1 and 0.8 as in HeartSteps. The second is a continuous choice of how large and how variable the next clawback is, with the total capped in advance.

That second action matters for two reasons. The finding in Eat & Tell that random losses beat fixed losses suggests penalty variance is something worth tuning. And continuous actions are where an actor-critic method has a structural advantage over greedy value-based methods, as discussed in Policy Gradient Methods for Reinforcement Learning with Function Approximation.

The prediction is specific. If frame-specific fatigue exists, meaning the loss frame's effect drops more steeply with its own exposure count than the gain frame's does, then the learned policy will send loss frames sparsely and concentrate them where sedentary risk is high. It should also keep its advantage after the money is withdrawn better than policies that use loss frames heavily.

Relation to Prior Work

Four mobile health papers supply the method and show that framing is missing. An Actor-Critic Contextual Bandit Algorithm for Personalized Mobile Health Interventions gives an online actor-critic with a least-squares critic and regret guarantees, but its action is only send or don't send, and its critic predicts immediate reward. Robust Actor-Critic Contextual Bandit makes that critic resistant to outliers with a capped loss, and Cohesion-based Online Actor-Critic pools parameters across similar users to shorten the cold start. We reuse both techniques, because loss-framed responses are likely heavy-tailed and early data is scarce. Personalized HeartSteps is the closest predecessor. It accounts for future responsiveness through a small proxy model and one dosage variable, but its messages are fixed activity suggestions. We differ by making the frame an action and giving each frame its own fatigue state. Optimizing Adaptive Notifications learns notification timing under a weekly budget with REINFORCE inside a simulator built from data, and it treats each notification as identical.

The behavioral side motivates the action space and constrains its design. Message framing to promote solar panels and BE IMMUNE show that loss and "reserved for you" frames raise one-shot uptake, but they assign frames statically. Eat & Tell shows that up-front endowments lose force over 30 days and that compliance relapses after incentives end. That relapse is why we measure behavior after withdrawal. The stickK analysis in Put Your Money Where Your Mouth Is shows that stake design affects habit survival over long horizons. Reinforcement Learning of Cooperative Persuasive Dialogue Policies using Framing already learns framing choices with reinforcement learning, but within a conversation and without sensing, fatigue, or money, so it does not cover the timing-plus-frame problem.

One result cuts against us. Impact of Push Notifications on Physical Activity and Sodium Intake found that pushes did not change short-term activity. A frame-aware policy cannot help if prompts carry no effect to begin with, so phase 1 estimates the effect of each frame directly.

Research Program

Phase 1 is a micro-randomized trial, a design in which each participant is re-randomized at many decision points. It runs eight weeks with roughly 80 sedentary adults, comparable in scale to the 82-person trial reported in Evaluation of the HeartSteps Online Sampling Algorithm. Decision points are anti-sedentary risk times, defined there as 40 or more sedentary minutes. At each one, the frame is randomized with fixed probabilities under an average budget of 1.5 messages a day. Each participant also holds a weekly endowed balance, with total possible loss capped in advance as in Eat & Tell. The penalty variance is randomized weekly. This phase answers the gating question: does the frame's effect on steps in the next 30 minutes interact with that frame's own exposure count more than with the exposure counts of other frames? We estimate this with the causal-excursion style of analysis that micro-randomized trials support. Because the randomization probabilities are recorded, the data also allows unbiased off-policy evaluation of candidate policies.

Phase 2 builds a simulator from the HeartSteps V1 generative model already used in Personalized HeartSteps and Cohesion-based Online Actor-Critic. We add frame-specific effects and $\lambda_f$ values fit to the phase-1 data. Since those values are uncertain, we also sweep them, including the case where every $\lambda_f$ is equal, so there is no frame-specific fatigue. The frame-aware actor-critic is developed and compared against baselines here.

Phase 3 deploys the chosen policy against the strongest baseline in a 12-week randomized comparison with roughly 120 participants. A 4-week follow-up without incentives follows.

Baselines

The naive reference point is a static design that sends a fixed loss-framed reminder at risk times under the same 1.5-per-day budget and charges a fixed clawback. This mirrors the deterministic "reserved for you" outreach of BE IMMUNE and the fixed-loss arm of Eat & Tell, alongside a no-message arm with the endowment only. On compliance, Eat & Tell reports final-day retention of 45.95% under random loss against 28.09% under fixed loss. Beating the static design means a higher mean 30-minute step response and higher retention at day 56 than this arm.

A second cheap baseline assigns frames uniformly at random and draws penalties randomly. This tests whether unpredictability alone, which Eat & Tell suggests can work, captures most of the benefit without any learning.

The strongest competitor is the Personalized HeartSteps algorithm, extended so that each frame is an arm. It is a Thompson-sampling bandit, meaning it picks actions by sampling from its current belief about each arm's reward. It keeps its proxy model of future responsiveness and its single dosage variable. It is strongest because it already accounts for delayed fatigue and has been deployed with people. In simulation it reports an average reward improvement of 29.753 over standard Thompson sampling, and its V2 pilot reports an average increase of 125 steps in the 30 minutes after a nudge (7 of 8 participants positive). We also run the myopic actor-critic of An Actor-Critic Contextual Bandit Algorithm with frame arms, to separate the value of the critic's view of the future from the actor-critic form itself.

Every method receives the same simulator draws, the same phase-1 warm-start data, the same budget, the same probability clipping, and an equal hyperparameter search budget of 50 configurations, scored on held-out simulated users.

Two ablations isolate the key ingredient. The first replaces the per-frame exposure counts with one pooled count. The second removes the penalty-variance action. Beating the extended HeartSteps baseline means a statistically reliable gain in cumulative step response over the 12 weeks and better retained activity after the money is withdrawn. Gains are reported only in the simulator settings with unequal $\lambda_f$, and we expect them to vanish when the $\lambda_f$ are equal.

Why not just…?

Why not just use a static loss frame, since it already works? The one-shot gains in the solar trial (5.32% against 4.17% for the earn frame) say nothing about the twentieth exposure. The Eat & Tell gap between fixed and random loss suggests that repetition erodes static loss schemes. The static arm is included so that this erosion is measured rather than assumed.

Why not just add frame arms to a contextual bandit like HeartSteps? This is the real threat. If fatigue is frame-blind, a single dosage variable captures it, and a bandit with frame arms should match our method. Phase 1 tests this directly through the frame-by-own-exposure interaction. The equal-$\lambda_f$ simulator condition checks that our method then loses its edge. If the interaction is near zero, we will report the bandit as sufficient and drop the actor-critic claim.

Why not just rotate frames at random, or by a fixed rule? Unpredictability is cheap, and Eat & Tell shows it helps, so the uniform-random arm may close much of the gap. It cannot target loss frames at high-risk moments or respond to a shrinking balance, which is where we expect the learned policy's advantage to concentrate. Fixed rules also failed under user heterogeneity: SeqRTS sent about 2 messages a day to sedentary users and about 1.1 to active ones. Off-policy evaluation of the random-rotation policy on phase-1 data settles this before deployment.

Why not REINFORCE, as in the notification simulator? With roughly five decisions a day, the variance of a policy gradient estimated without a critic is the binding constraint. We include a REINFORCE variant in the simulator, at matched sample counts, to check this rather than assume it.

Decisive Checks

The project lives or dies on two results. The first is whether phase 1 detects a frame-specific fatigue interaction whose confidence interval excludes zero. The second is whether, in phase 3, the frame-aware actor-critic beats frame-armed HeartSteps on cumulative 30-minute step response and on activity during the 4-week period after incentives end, while staying within 1.5 ± 0.15 messages a day.

A third check guards against a learner that wins by pushing loss frames harder. We report the fraction of decisions that use a loss frame, and app-uninstall rates, for every policy. Clawbacks are capped in advance and the protocol requires ethics review. The behavioral claim is supported only if the learned policy uses loss frames less often than the bandit while achieving more.