Early Action Prediction with Drifting Personal Priors

When a robot watches a person begin to move, the first fraction of a second rarely says what the person is about to do. Reaching for a cup and reaching for a spoon start the same way, and gestures begin from the same resting pose. Early action prediction systems, which classify an action from only its first 10–30%, try to squeeze the answer out of those ambiguous first frames. Yet a person's habits carry much of the missing information. Someone who usually drinks after eating is probably reaching for the cup. Those habits do not stay fixed. They change with fatigue, with familiarity with the robot, and with the task of the day. This proposal treats early prediction as two problems: reading the motion, and maintaining an up-to-date, per-person belief about what that person tends to do next. It argues that tracking how this belief drifts matters most exactly when the motion is least informative.

The concrete idea is to factor the early prediction into a shared motion model and a per-user prior over actions. The motion model reports how well the partial movement fits each action. The prior is updated online from the labels the system receives for free each time an action completes. A changepoint tracker tells the system when the person's habits have shifted, so it does not need a hand-set forgetting rate. The central testable claim is narrow: the benefit of tracking a drifting prior should be large at 10–20% observation, should shrink smoothly as the motion becomes informative, and should be near zero for full actions. If the gain is uniform across observation ratios, or absent at early ratios, the idea is wrong.

Core Claim

At observation ratio $\tau$, the predicted action for user $u$ at interaction index $t$ is

$$ p(a \mid x_{1:\tau}, u, t) \;\propto\; \frac{q_\theta(a \mid x_{1:\tau})}{\pi_{\text{train}}(a)}\,\pi_{u,t}(a). $$

Here $q_\theta$ is any early action classifier trained across all users. Dividing by its training class frequencies $\pi_{\text{train}}$ turns its output into a quantity proportional to a likelihood. $\pi_{u,t}$ is the user's current prior over actions. When the observed motion $x_{1:\tau}$ is short, the likelihood ratio is nearly flat across actions that share a kinematic prefix, so the posterior is governed by $\pi_{u,t}$. A stale prior then costs the most accuracy at small $\tau$. The prior's influence falls as the likelihood sharpens, which is why we predict a gain profile that decays with $\tau$.

The two literatures leave this open from opposite sides. Early prediction methods such as Early Action Prediction by Soft Regression (soft, category-specific confidence labels that sharpen with progress), Progressive Teacher-Student Learning for Early Action Prediction. (distilling a full-video teacher into a prefix-only student), and ERA: Expert Retrieval and Assembly for Early Action Prediction (routing prefixes to specialized expert kernels) all assume one fixed mapping from prefix to label, shared by everyone, with no notion of who is moving or when. We reuse them unchanged as $q_\theta$; the contribution sits outside them. The drift literature motivates the prior side. Right Now, Wrong Then: Non-Stationary Direct Preference Optimization under Preference Drift shows that a stationary learner falls to near-chance when preferences flip, and that exponential recency weighting repairs much of the damage. It also notes that the right discount depends on an unknown amount of drift. That point constrains our design: we replace a fixed discount with an inferred changepoint. Fast Motion Prediction for Collaborative Robotics already combines a Bayesian classifier with early reaching motion. It uses a fixed library and a fixed, implicitly uniform prior, which is the specific assumption we relax.

Research Program

The motion model is trained once on pooled data and then frozen. This choice follows Personalization in Human-Robot Interaction through Preference-based Action Representation Learning, whose evidence that fine-tuning a shared policy on per-user feedback destroys task competence (feeding success 88% versus 41%) argues for putting all personalization in a small separate component. We calibrate $q_\theta$ separately at each observation ratio with temperature scaling on held-out data, because the prior correction is only as good as the likelihood calibration.

The per-user prior is a Dirichlet over actions. It is updated by the completed-action label after each interaction, so it needs no human feedback beyond what normal operation produces. Drift is handled by maintaining a distribution over "how many interactions ago this user's habits last changed." Each possible age keeps its own Dirichlet counts, and a constant hazard rate sets how often changes are expected. The prior $\pi_{u,t}$ is the average of these candidate priors, weighted by the current belief about which age is correct. When completed actions start surprising the current prior, belief moves toward a recent changepoint and old counts are dropped automatically. Gap 3 of the research file, which separates enduring from transient preferences, suggests one refinement. The prior can mix a slowly updated long-term component with the changepoint-reset short-term component, so a long-standing habit is not discarded by a brief deviation. We treat this as an ablation, not a core claim.

We also add a minimal hedge against the progress-level problem raised in Early Action Prediction by Soft Regression. Fatigue-related slowing can make a prefix look earlier than it is. We therefore evaluate both with given progress ratios and with elapsed time since motion onset. Within-action goal changes, the kind of drift highlighted in the file's Gap 1, are out of scope. Our drift operates between actions, which is where the per-user prior lives.

The benchmark needs action sequences grouped by person with a known drift structure, and no listed dataset has both. We build them the way Right Now, Wrong Then: Non-Stationary Direct Preference Optimization under Preference Drift builds drift into UltraFeedback: take real data and impose controlled change points. On NTU RGB+D and SYSU 3DHOI, each pseudo-user is a stream of held-out sequences whose class frequencies follow a user-specific prior. That prior changes by sudden flips among actions with shared prefixes, by gradual rotation, or not at all. On the tabletop reaching data of Fast Motion Prediction for Collaborative Robotics, which has 12 classes built from three shared start positions and four targets, the drifting quantity is the preferred target. In Assistive Gym: A Physics Simulation Framework for Assistive Robotics, simulated humans switch between activities of daily living on a schedule tied to simulated fatigue. Finally, the 15-participant handover recordings of Early Detection of Human Handover Intentions in Human-Robot Collaboration: Comparing EEG, Gaze, and Hand Motion provide a small real-person check on whether natural within-session drift in handover frequency exists at all.

Baselines

The naive reference is the frozen shared classifier used as published, with no user information. We use the strongest model we can reproduce from Progressive Teacher-Student Learning for Early Action Prediction., which reports an AUC of 60.97% on NTU RGB-D, and ERA, which reports more than an 11% accuracy gain over HARD-Net at 20% observation on NTU60 but gives no absolute figure in the file. On stationary pseudo-users this baseline should tie our method. On drifting users it sets the floor. Any gain must come from the prior, because the classifier is identical in every arm.

The strongest competitor for the drift side is recency-discounted counting in the spirit of NS-DPO. The per-user prior is built from completed labels weighted by $\gamma^{T-t_i-1}$, applied to the same frozen classifier through the same equation. We tune $\gamma$ by grid search on validation users, with the same number of validation streams as we use to set our hazard rate. We also report an oracle-$\gamma$ version tuned on each test drift regime separately. No reported result was found on Lacuna for recency-weighted priors in early action prediction. The closest reported figures are NS-DPO's alignment numbers: reward accuracy above 60–70% where DPO fell to roughly 50–55%, in a different domain. Beating this baseline means higher early-ratio accuracy, averaged over a mixture of drift regimes, than the single best validation-tuned $\gamma$, without the result depending on knowing the regime.

We include a static per-user prior estimated from each user's full history. This is the "persona embedding" idea reduced to its essence, and it isolates the value of tracking drift as opposed to merely personalizing. We also include an oracle that knows the true current prior. Its gap above our method measures the remaining drift regret.

Two ablations isolate the key ingredients. The first removes the likelihood correction (multiplying raw $q_\theta$ by the prior), which tests whether calibration-correct factoring matters. The second removes the slow long-term component. All arms share data, the frozen backbone, the calibration split, and an equal hyperparameter-search budget.

Why not just…?

Why not just feed the user's recent action history into the classifier as extra input? A sequence model over past actions plus the current prefix could learn habit structure implicitly. This is a real threat for short-range regularities, such as "drink follows eat." It needs per-user training data or meta-training over many simulated users. It also has no explicit way to discard old evidence after a long-horizon change. We train this model on the same pseudo-user streams, with history windows of 5, 20, and 100 actions, and compare it with ours. The prediction is that it matches us on stationary or short-cycle users and trails after sudden flips, with slower recovery. If it matches us everywhere, we will report that the explicit factorization buys interpretability and data efficiency but not accuracy.

Why not just use a fixed forgetting rate, as NS-DPO does? This is the most serious alternative, and it is why the experiments mix drift regimes. When a single regime is known, a well-tuned $\gamma$ may match a changepoint tracker. NS-DPO itself states that the right $\gamma$ depends on an unknown variation budget, and our claim rests on that unknown. If validation-tuned $\gamma$ matches our method on the mixed-regime test set, we will drop the changepoint machinery. We would then present the contribution as the factored prior plus the early-ratio gain profile, which would still hold.

Why not just fine-tune the classifier online on each user's recent actions? This folds personalization into the motion model. The direct evidence from Personalization in Human-Robot Interaction through Preference-based Action Representation Learning is that this kind of fine-tuning erodes shared competence. That evidence comes from control, not classification, so we test the point directly. We run per-user online fine-tuning of the last layer with replay and measure accuracy both on the user's stream and on a fixed population test set, which detects forgetting. Our method cannot forget, because the classifier never changes.

Decisive Checks

The first check is the gain profile. We plot accuracy gain over the naive baseline against observation ratio from 10% to 100% on NTU RGB+D, SYSU 3DHOI, and the reaching data. The claim survives only if the gain is largest at 10–20% and shrinks toward zero at 100%. We additionally expect the gain to be largest for class pairs with shared prefixes, which we identify by the confusion matrix of the frozen classifier at 10%.

The second check is recovery after a sudden flip. We measure the number of completed actions until early-ratio accuracy returns to within a fixed margin of the oracle, comparing our method with the tuned-$\gamma$ and action-history baselines. The third is the mixed-regime comparison against the best single $\gamma$, which decides whether the changepoint tracker earns its place. The fourth is a stationary control, where our method must not lose to the naive baseline; this mirrors NS-DPO's no-penalty result under stationarity.

On the handover recordings, we first test whether the frequency of handover versus non-handover intent drifts within a session. If it does, we ask whether the drifting prior moves detection earlier relative to movement onset, against the roughly one-second-before-onset figure the paper reports for its multimodal fusion. If no drift exists in that data, we report so, and the real-world claim stays limited to simulation and semi-synthetic streams.