Learning When, Not How Often: Bounded Residual Policies for Just-in-Time Health Nudges

Mobile health apps increasingly try to send a message, such as a prompt to walk after a long sedentary stretch, at the moment it is most likely to help. They also try to stay quiet when a message would only annoy. These systems are called just-in-time adaptive interventions (JITAIs). Reinforcement learning (RL) looks like the natural way to learn such timing, and in simulation it works. In the StepCountJITAI environment, deep RL agents reach returns around 3,000, while Thompson Sampling bandits reach around 1,500 (StepCountJITAI). Deployed trials, however, still run hand-designed randomization rules. The obstacle is not performance at convergence. End-to-end agents need very many episodes to learn, and they explore by trial and error. On a real person, that exploration means bursts of messages, habituation, and people deleting the app. RL also offers no way to promise a clinician that the patient will receive roughly the prescribed number of messages per day.

We propose to split the decision in two. A clinical base rule, such as the budgeted sampler used in the HeartSteps trial, keeps deciding how many messages a person receives. A small learned residual policy decides when those messages go out. It can only rescale the base rule's delivery probability within a fixed multiplicative band, and its input is its current estimate of the user's context and hidden state. Residual policy learning means learning a correction on top of an existing controller rather than a policy from scratch. It is well established in robotics but has not been tried in behavioral systems. Carried into this setting, the bounded form gives three things at once. The dose guarantee holds from the first day of learning. Every decision point keeps a nonzero chance of either action, so trial data still supports causal analysis. And any learned policy can be evaluated offline on existing trial data with bounded importance weights, because it can never stray far from the policy that collected the data.

Core Claim

Let $p_{\text{base}}(s_t)$ be the probability that the clinical rule sends a message at decision point $t$. It depends on the current risk-time status and on how much of today's budget remains. Let $b_t$ be the agent's belief over the user's context, habituation, and disengagement risk. The residual policy $f_\theta$ outputs a value in $[-1,1]$, and the delivered probability is

$$p_t = \mathrm{clip}\big(p_{\text{base}}(s_t)\,(1 + \beta f_\theta(b_t)),\; p_{\min},\; p_{\max}\big),$$

where $\beta \in [0,1)$ sets the width of the band and $[p_{\min}, p_{\max}]$ keeps randomization strictly inside $(0,1)$.

This is the relative "safety tube" of constrained residual RL for mechatronics. That work bounded a continuous torque correction around a PI controller and proved stability. Here the same shape is applied to a Bernoulli delivery probability.

Two facts carry the proposal:

  1. The dose guarantee survives learning. The base sampler's budget tracker counts messages actually delivered, not messages it intended to send. The residual can therefore shift probability mass between risk times within a day, but it cannot push the daily total far above target. That total stays within a factor of $(1+\beta)$ of what the base allows at each point.
  2. Offline evaluation stays low-variance. The per-step likelihood ratio between the residual policy and the logging policy is bounded by $\max\!\big(1+\beta,\ (1-p_{\min})/(1-p_{\max})\big)$. Importance-weighted evaluation on micro-randomized trial data (trials that randomize the intervention at every decision point) therefore has controlled variance, which unconstrained end-to-end policies cannot offer.

The claim to test is this: within this band, a residual that conditions on uncertain context and on inferred hidden states recovers most of the gap between the base rule and unconstrained deep RL. It should do so with far fewer episodes and with near-zero burden violations during training.

Why This Is Different From Existing Work

Prior JITAI learning comes in two families.

The first uses heuristics and bandits. The HeartSteps bandit uses Thompson Sampling with a decaying dosage term and clips probabilities to 0.1–0.8 (Personalized HeartSteps). The SeqRTS sampler hits an average of 1.486 messages per day against a 1.5 target (HeartSteps online sampling). Both methods respect burden, but neither learns long-horizon timing. SeqRTS also over-treated sedentary users at about 2 messages per day and under-treated active users at about 1.1. This subgroup skew is our main motivating evidence. Keep the aggregate budget, but let learning fix the per-person allocation.

The second family trains end-to-end policies:

  • REINFORCE, PPO, and DQN in StepCountJITAI (StepCountJITAI).
  • REINFORCE versus Dueling DQN under context inference error (Marlin et al.). That study showed that feeding the agent context probabilities rather than hard labels adds roughly 500–800 reward units at an 18% error rate, and that value-based methods lose about 50% of their performance when hidden states are unobserved.
  • REINFORCE under a hard cap of 14 notifications per week in a behavioral simulator (Wang et al.).

None of these methods anchors to a clinical rule. None bounds behavior during learning. None considers offline evaluation on trial logs.

We reuse two findings from Marlin et al. as design choices. The residual receives the probability vector rather than a label, and it is trained with policy gradients rather than value-based methods.

Safe-RL methods constrain the design without supplying it directly. Backward value functions give constraint satisfaction during training, but they assume an ergodic, stationary process and a feasible starting policy. Human behavior violates the first two assumptions. We instead obtain feasibility structurally: the base rule is feasible by construction and the band preserves feasibility. Feasible dual policy iteration reports that safer policies see fewer violations, which makes estimating the constraint unstable. Our design never needs to estimate violations for the dose constraint. Only the softer disengagement constraint is handled by a learned multiplier.

Research Program

The formulation. The base is a reimplementation of SeqRTS-style budgeted sampling. We treat the dose constraint as structural. We treat disengagement risk as a constrained-MDP cost (a separate cost signal with a budget, rather than a reward penalty). Its multiplier is updated by the PID-Lagrangian scheme that Safety Gymnasium found suppresses the oscillations of plain Lagrangian updates. The residual is a small recurrent network over $b_t$, trained by a clipped policy gradient.

The environment. StepCountJITAI does not natively include daily budgets or risk times. We will extend it with a sedentary-risk indicator and a daily budget. We will also add a heterogeneous population whose habituation and disengagement parameters are drawn per user, spanning "sedentary" and "active" types. The added heterogeneity is what lets us reproduce the SeqRTS subgroup skew.

The analysis. We will state the dose bound and the importance-weight bound formally. We will then measure empirically how the achievable return varies with $\beta$. This gives clinicians a knob with a known meaning: at most a $\beta$ fraction of deviation from the protocol at any decision point.

The reward shaping, a secondary question. The multi-objective literature argues that additive reward terms get gamed, while multiplicative composition forces an agent to satisfy every term (SkillMimic, redundancy resolution). We test one version of that claim. We compare two options inside the residual: an additive step-minus-burden reward, and a product of the step response and a retention probability. Our expectation, stated honestly, is that once dose sits in the structure and disengagement sits in a constraint, the shaping choice matters little. Finding that would itself be useful guidance.

Baselines

Base rule alone. This is the do-nothing reference: the SeqRTS-style sampler with its published parameters (HeartSteps online sampling). It fixes the floor for return and the reference for dose. The residual method beats it if it raises return while keeping the daily dose within the $(1+\beta)$ envelope.

Per-subgroup tuned base rule. This is the strongest cheap competitor. We grid-search the sampler's parameters separately for each simulated user type. The grid gets the same total number of simulator episodes the residual uses for training.

Thompson Sampling with dosage and clipping. This is the HeartSteps bandit (Personalized HeartSteps). It is the deployed-in-practice learner, and StepCountJITAI reports it at around 1,500.

Unconstrained end-to-end PPO and REINFORCE. These receive probabilistic context input and the same episode budget. StepCountJITAI reports around 3,000 for these agents under observed $[C,H,D]$. That is the ceiling we aim to approach, not necessarily exceed.

End-to-end constrained RL. These are PPO-Lagrangian and CPPO-PID from Safety Gymnasium, and FOCOPS, each with dose and disengagement expressed as expected-cost constraints. This is the strongest existing method on the actual question of safe learning. Its reported numbers come only from physical tasks: FOCOPS reaches 1106.1 reward on Humanoid-Circle versus 671.0 for CPO. We found no reported result on Lacuna for constrained RL in a JITAI simulator. We will therefore run it ourselves, with the same simulator, episode budget, network size, and hyperparameter search budget as our method.

Ablations. - An unbounded residual ($\beta$ large, no clipping) separates the effect of the band from the effect of anchoring. - A residual that sees labels rather than probabilities separates the effect of uncertain context input. - A residual that sees context but not the belief over hidden states separates the effect of hidden-state inference.

What counts as winning. Beating the constrained end-to-end methods means one of two outcomes at the same final return: fewer cumulative dose or disengagement violations during training, or reaching 90% of their final return in substantially fewer episodes.

Why not just…?

Why not just tune the clinical rule per user type? This is the real threat. The SeqRTS failure was a mis-allocation across subgroups, and per-subgroup tuning might fix most of it. The difference is that a tuned rule cannot react within a day to a belief that habituation is rising or that the context is uncertain. Marlin et al. show that exploiting uncertain context is worth hundreds of reward units. The tuned-rule baseline, run with a matched episode budget, settles the question. Suppose the residual's advantage over it falls within seed variance under both low and high context error. We would then report that the residual's value lies only in offline evaluation and in personalization within a type, and we would reframe the contribution accordingly.

Why not just use the HeartSteps Thompson Sampling bandit? It already respects burden through clipping and was deployed. However, it optimizes one decision at a time, and StepCountJITAI reports it at roughly half the return of RL agents. Our experiments test whether the residual closes that gap under the same clipping range.

Why not just train constrained PPO or FOCOPS end to end? Constrained PPO or FOCOPS satisfies constraints at convergence rather than during learning. Safety Gymnasium also shows that PPO-Lagrangian gives up 45% of reward to reach its violation reduction. The training-time violation curves and episodes-to-threshold measurements in our first experiment test this directly. If CPPO-PID matches our training-time violation rate at equal sample cost, then the band adds only the offline-evaluation benefit, and we will say so.

Why not just subtract a burden penalty from the reward? Additive penalties require tuning a weight that swings between ignoring burden and never messaging, as MetaDrive and Safety Gymnasium document. We include a penalty-weight sweep as an end-to-end baseline to show where that failure appears in the JITAI setting.

Proposed Experiments

Experiment 1: learning efficiency and training-time safety. In the extended StepCountJITAI, we sweep $\beta \in \{0.25, 0.5, 1\}$ and context error rates spanning the 10–41% range of Marlin et al. We report the following, over 20 seeds:

  • final return;
  • episodes needed to reach 90% of the unconstrained PPO return;
  • cumulative days over budget during training;
  • simulated disengagement events;
  • the dose ratio between sedentary and active users, benchmarked against the reported SeqRTS skew of about 2 versus 1.1 messages per day.

The decisive check is whether the residual beats the tuned-rule and CPPO-PID baselines on return per violation over the full training curve, not just at the end.

Experiment 2: offline evaluation on trial logs. We use the HeartSteps V2V3 micro-randomized trial logs: 82 participants and 10,137 messages across 166,742 risk times. This assumes the per-decision randomization probabilities were logged, which is standard for such trials. We train residual policies in simulation and evaluate them on the logs with weighted importance sampling. We compare effective sample size and confidence-interval width against evaluating end-to-end PPO on the same logs. The prediction is that PPO's weights collapse onto a handful of decision points, while the residual keeps an effective sample size proportional to $1/(1+\beta)^2$ per step.

Experiment 3: reward shaping inside the residual. We run the additive versus multiplicative comparison, measuring return and disengagement. Two findings would be decisive: shaping matters only when the structural bound is removed, or it matters even with the bound in place.